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Executive  Summary 

The  Commander  in  Chief  21st  Century  (CINC  21)  is  an  Advanced  Concept  Technology 
Demonstration  (ACTD)  that  was  designed  to  improve  command  and  control  and  decision¬ 
making  through  network-centric,  knowledge-enabled  staff  functions.  CINC  21’s  goal  is  to 
improve  command  and  control  of  forces  by  exploiting  advanced  visualization  techniques  and 
decision  support  systems  (e.g.,  presentations,  cueing,  triggers,  alerts),  collaboration  capabilities, 
and  knowledge  and  enterprise  management  technologies.  CINC  21  used  information  technology 
to  reduce  dependence  on  centralized,  single-crisis  command  centers  in  favor  of  decentralized, 
rapidly  configured  coalition  and  interagency  enclaves. 

To  evaluate  the  technologies  developed  as  part  of  the  CINC  21  ACTD,  a  Joint  Military  Utility 
Assessment  (JMUA)  was  planned  and  executed.  The  purpose  of  the  JMUA  was  to  determine 
how  well  the  CINC  21  products  met  warfighter  requirements. 


What  is  CINC  21? 


The  vision  of  CINC  21  is  to  develop  and  assess  a  decision-focused  command  and  control 
framework  that  enables  joint  commands  to  dynamically  manage  and  focus  organization, 
processes,  resources,  and  information  to  improve  situational  awareness  and  decision-making. 
CINC  21  comprises:  an  advanced  concept  of  a  synchronized  battle  rhythm  and  an  enterprise 
architecture,  an  integrating  framework  for  decision-focused  command  and  control,  a  set  of 
integrating  services;  a  set  of  applications  (operational  packages),  command  and  control 
infrastructure  services,  hardware  displays,  computer  hardware,  and  security  systems. 


Why  Develop  CINC  21? 


The  operational  concept  for  CINC  21  is  a  knowledge-enabled  information  sphere  with  tools  and 
applications  to  (1)  improve  situational  awareness  and  understanding,  (2)  facilitate  the  ability  to 
collaborate  when  necessary,  and  (3)  manage  the  information  enterprise,  while  transforming  and 
accelerating  the  decision  processes  that  underlay  the  management  of  crisis-contingency 
operations,  theater  engagement,  and  staff  processes.  The  critical  operational  issues  (COIs)  for 
CINC  21  were: 

•  Can  advanced  visualization  technology  empower  individuals  to  process,  digest,  and 
assimilate  large  volumes  of  information,  enabling  faster,  more  effective  decisions? 

•  Can  knowledge  management  technology  integrate  information,  context,  and  rules  to 
increase  understanding  and  improve  decision-making? 

•  Can  collaboration  tools  be  used  to  overcome  tyrannies  of  time,  distance,  and  system 
disparity? 

•  Can  the  collection  of  networks,  databases,  and  applications  be  enhanced  to  optimize  the 
flow  of  information,  with  security  assurance,  across  multiple  network  enclaves? 
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Who  Had  Assessment  Responsibility? 


The  principal  organizations  that  executed  CINC  21  were  the  Deputy  Under  Secretary  of  Defense 
(DUSD)  Advanced  Systems  &  Concepts  (AS&C),  United  States  Pacific  Command 
(USPACOM),  U.S.  Strategic  Command  (USSTRATCOM),  the  Office  of  Naval  Research 
(ONR),  the  Defense  Information  Systems  Agency  (DISA),  Space  and  Naval  Warfare  Systems 
Command  (SPAWAR),  DUSD  (Science  &  Technology),  the  Defense  Intelligence  Agency 
(DIA),  and  the  Defense  Threat  Reduction  Agency  (DTRA).  USPACOM  was  the  sponsoring 
Combat  Commander  (COCOM)  and  was  the  Operational  Manager  for  the  Advanced  Concept 
Technology  Demonstration  (ACTD).  The  Office  of  Naval  Research  (ONR)  and  Defense 
Information  Systems  Agency  (DISA)  shared  responsibilities  of  Technical  Manager.  SPAWAR 
acted  as  the  Transition  Manager  for  the  CINC  21  ACTD.  USPACCOM  and  United  States 
Strategic  Command  (USSTRATCOM)  provided  the  operational  settings  for  the  ACTD. 


How  Was  CINC  21  Assessed? 


Based  on  the  COIs  and  prioritized  requirements,  a  Joint  Military  Utility  Assessment  (JMUA; 
Appendix  B)  plan  was  developed  to  evaluate  CINC  2 1  capabilities  for  acquiring,  processing, 
analyzing,  and  presenting  information  needed  for  better  decision  making,  improved  processes, 
increased  overall  readiness,  efficiency,  and  mission  effectiveness.  Measures  of  effectiveness 
(MOEs)  were  defined  along  with  subsequent  measures  of  performance  (MOPs)  and  metrics  to 
assess  the  CINC  2 1  products  in  an  operational  setting  and  ensure  validity  of  measurement.  These 
measures  and  metrics  were  organized  by  three  mission  effectiveness  criteria:  Suitability, 
Usability,  and  Technical. 

CINC  2 1  assessment  efforts  were  conducted  throughout  the  technology  development  and 
integration  period.  Consequently,  different  sets  of  assessment  methods  and  metrics  from  the 
JMUA  Plan  were  used  as  appropriate  for  the  technology  type,  development  maturity,  and 
assessment  environment.  Trouble  reports  and  technical  observations/logs  were  analyzed  to 
identify  successes  and  problems  associated  with  the  installation  and  integration  of  the  diverse  set 
of  CINC  21  products.  The  usability  of  CINC  21  products  was  assessed  by  several  methods:  (1) 
heuristic  evaluations  that  compared  the  user  interface  against  industry  standards  and  guidelines, 
(2)  infonnal  comments  and  observations  by  representative  users,  and  (3)  structured  ratings  and 
perfonnance  tests  of  the  user  interface  under  scenario-based  task  conditions.  The  impact  of 
CINC  2 1  products  on  organizational  effectiveness  and  work  processes  was  assessed  in  similar 
ways.  The  suitability  of  CINC  2 1  technology  to  support  operational  tasks  was  assessed  via  direct 
involvement  by  representative  operational  decision-makers  and  staff.  They  provided  comments 
and  ratings  on  the  technology  during  several  demonstrations  and  training  sessions,  and  they  used 
the  technology  for  their  dynamic  tasks  in  large-scale  exercises. 
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Outcome 


The  composite  ratings  for  Suitability,  Usability,  and  Technical  for  each  of  the  CINC  21 
technology  categories  are  summarized  below  in  the  table  and  accompanying  text.  The 
technology  category  ratings  are  given  for  the  COIs  that  are  applicable.  In  cases  where  a 
technology  category  applies  to  more  than  one  COI,  different  aspects  were  evaluated.  For 
example,  Decision-Focused  Command  and  Control  (DFC2)  involves  the  presentation  of 
information  (COI-1)  and  the  organization  and  management  of  that  information  in  a  common 
knowledge  core  (COI-2). 

Decision-Focused  Command  and  Control  (DFC2) 

The  concept  of  DFC2 — integrated  applications  that  facilitate  the  definition,  visualization, 
analysis,  and  management  of  decision-making  activities  associated  with  joint  command  and 
control — was  well  received  by  users.  However,  there  is  currently  a  lack  of  functioning  links 
between  component  parts  and  between  decision  objects  within  component  parts.  The  document 
linking  and  management  services  in  their  current  stage  of  development  do  not  fulfill  users’ 
needs.  In  addition,  concerns  regarding  the  impact  of  DFC2  upon  workload  and  organizational 
procedures  remain.  Usability  problems  were  experienced  throughout  the  assessment  events, 
especially  in  areas  of  configuration  retention  and  navigation.  Excessive  use  of  scrollbars  limit 
users’  ability  to  maneuver,  and  they  present  a  confusing  display.  Many  functions  require 
numerous  steps  as  well  as  interaction  with  multiple  dialog  boxes.  The  use  of  screen  real  estate  is 
less  than  optimal;  web  parts  are  located  partially  or  completely  off  the  screen.  A  slow  refresh  rate 
and  some  instability  of  component  parts  occurred  throughout  the  assessment  events.  These 
problems  interfered  with  users  as  they  attempted  to  enter  or  access  information.  There  were 
intermittent  crashes  of  DFC2  during  TF03;  these  crashes  were  associated  with  data  losses. 

Operational  Packages 

The  Operational  Packages  are  applications  that  use  the  underlying  enterprise  architecture  to 
provide  new  production  capabilities  for  staff  and  subordinate  commands.  These  applications 
provided  information  that  is  stored  in  databases  and  is  then  available  as  supporting  content  that 
can  be  aggregated/rolled  up  for  the  decision-maker.  The  following  sections  describe  the 
operational  packages. 

Common  Operational  Picture/Geo-Spatial  Visualization 

The  Area  of  Responsibility  Basing  (AORB)  and  Fused  Battlespace  View  (FBV)  tools  helped 
users  perform  their  assigned  tasks  and  achieve  their  operational  goals.  Users  appreciated  the 
concept  of  the  geo-spatial  representation,  in  which  they  could  drill  down  and  gather  their  desired 
content.  Users  successfully  navigated  with  the  tools  and  reached  their  target  information.  The 
technical  implementation  of  these  tools  was  less  than  satisfactory,  however.  AORB  was  in  an 
early  stage  of  development  and  the  databases  needed  to  support  its  capability  were  not  available. 
FBV  did  not  yet  include  classified  information,  which  users  said  was  needed  for  it  to  be  of  real 
value. 
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Technology 

Category 

COM 

COI-2 

COI-3 

001-4 

S 

U 

T 

S 

U 

T 

S 

U 

T 

S 

U 

T 

Decision- 
Focused  C2 

A 

A 

A 

A 

A 

A 

NA 

NA 

NA 

NA 

NA 

NA 

Ops  Package: 

COP/Geo-Spatial 

Visualization 

© 

© 

A 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

Ops  Package: 

Briefing/ 

Information 

Summary 

© 

© 

© 

© 

© 

A 

NA 

NA 

NA 

NA 

NA 

NA 

Ops  Package: 

Status  Tracking 
and  Linking 

© 

A 

© 

A 

A 

A 

NA 

NA 

NA 

NA 

NA 

NA 

Ops  Package: 

Time-Based 

Event 

Management 

© 

© 

© 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

Ops  Package: 

JWID  Coalition 
Interoperability 
Service  (COINS) 

NA 

NA 

NA 

© 

A 

© 

NA 

NA 

NA 

NA 

NA 

NA 

Displays: 

Group  Displays 

© 

© 

© 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

Displays: 

Workstation 

Displays 

© 

© 

© 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

Collaboration 

NA 

NA 

NA 

NA 

NA 

NA 

© 

A 

A 

NA 

NA 

NA 

Network 
Operations 
Tools  and 
Services 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

© 

A 

A 

Note:  S  =  Suitability,  U  =  Usability,  and  T  =  Technical. 
Note:  G  g  Green,  Y  =  Yellow,  and  R  =  Red. 
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Briefing/Information  Summary 

The  COA  Matrix  and  Consequence  Management  Automated  Brief  tools  appear  to  meet  their 
operational  requirements.  There  are,  however,  many  usability  issues  that  should  be  improved. 

The  usability  of  the  COA  Matrix  was  well  received  by  the  operators.  The  usability  problems  with 
Consequence  Management  should  be  addressed  before  further  development. 

Status  Tracking  and  Linking 

The  Message  Tracker  is  effective  at  linking  messages  to  significant  events  and  increasing  staffs 
access  to  information.  Improving  its  interface  features,  error  prevention  and  detection,  online 
help,  and  searching  will  enhance  its  usability.  However,  none  of  these  shortcomings  prevent 
users  from  accomplishing  their  tasks.  The  RFI  Manager  had  significant  problems  with  Usability 
and  Suitability  and  will  require  redesign  to  become  an  effective  tool.  Aside  from  slow  response 
times  in  some  circumstances,  TeamApp  performs  its  functions  well.  It  simplifies  the  insertion  of 
documents  into  Web  pages,  facilitates  staff  interaction,  and  may  also  reduce  manpower 
requirements. 

Time-Based  Event  Management 

While  there  is  room  for  improvement  in  addressing  the  needs  of  individual  users,  the  Master 
Calendar  and  the  Task  Management  System  (TMS)  generally  meet  the  needs  of  their  users.  They 
have  been  accepted  by  the  users  and  are  currently  being  used  at  USPACOM. 

JWID  Coalition  Interoperability  Service  (COINS) 

JWID  COINS  was  judged  to  provide  capabilities  of  value  for  integrating  information,  context, 
and  rules  to  increase  understanding  and  improve  decision-making.  The  ease  of  use  and  display 
issues  with  this  technology  are  problematic  and  may  affect  user  acceptance.  Problems  noted  with 
the  human-computer  interface  design  for  data  entry,  displayed  results,  and  multi-step  processes 
need  to  be  resolved.  Further  development  effort  is  needed  to  provide  a  satisfactory  refresh  rate 
and  to  provide  the  ability  to  save  inputs. 

Displays 

Group  Displays 

The  USSTRATCOM  Knowledge  Wall  and  the  USPACOM  Video  Wall  are  currently  in  use, 
function  reliably,  and  generally  meet  the  needs  of  their  users.  The  USSTRATCOM  Knowledge 
Wall  effectively  met  the  needs  of  the  senior  decision-makers  during  the  October  2002 
demonstration.  Knowledge  Wall  operators  were  able  to  manipulate  content  and  present  it  in  a 
way  that  was  relevant  and  legible.  There  are  some  unresolved  issues  regarding  the  military  utility 
of  the  Video  Wall.  The  success  of  the  Knowledge  Wall,  though,  outweigh  the  inadequacies 
found  on  the  Video  Wall,  which  is  not  making  the  transfer  to  the  new  command  center. 

Workstation  Displays 

The  Workstation  Displays,  consisting  of  multiple  side-by-side  monitors  and  a  single  multi-panel 
display  console  (PV290),  were  well  received  by  users.  The  PV290  perfonned  without  incident 
during  the  USSTRATCOM  October  2002  demonstration.  During  TF03  at  USPACOM,  users 
reported  their  three-screen  flat-panel  display  workstation  was  essential  in  providing  users  a 
continuous  view  of  the  various  web  parts.  This  view  increased  the  usability  of  the  displays  and 
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helped  meet  their  operational  requirements.  However,  the  workstation  was  effective  only  in 
presenting  content  to  the  user  sitting  directly  in  front  of  the  screens  and  not  to  others  standing 
around  the  display. 

Collaboration 

Distributed  users  found  high  value  in  sharing  documents,  briefs,  and  desktops.  They  considered 
that  collaboration  tools  increased  situation  awareness  by  providing  an  efficient  mechanism  for 
the  exchange  of  infonnation.  However,  the  inability  of  CINC  2 1  collaborative  services  to 
provide  stable  connectivity  is  a  major  obstacle  toward  reaching  the  goal  of  synchronizing 
decision-making  and  information  management  among  distributed  users.  Other  impediments  to 
use  are  lack  of  training,  lack  of  business  rules,  and  lack  of  interoperability  between  collaboration 
systems.  Chat  often  became  the  de  facto  collaboration  tool  due  to  bandwidth  restrictions.  Audio 
was  preferred  when  available;  video  was  often  unstable  and  considered  by  many  users  to  be 
redundant  and  unnecessary.  The  status  of  audio-video  and  of  session  membership  was  frequently 
difficult  to  discern.  In  addition,  feedback  to  the  user  regarding  success  of  data  transmittals  was 
not  always  adequate. 

Network  Operations  Tools  and  Services 

Many  network  operations  tools  and  services  were  implemented.  These  tools  give  the  network 
managers  the  ability  to  interact  with  other  management  tools.  The  portal  approach  resulted  in 
flexibility  of  display  and  screen  space  usage.  The  users  found  the  Theater  C4ISR  Status  Grid  to 
be  useful  for  such  tasks  as  changing  layout  and  exchanging  information.  However,  workload  was 
not  decreased,  and  learning  how  to  use  the  capability  was  difficult.  Transmitted  information  did 
not  increase  the  ability  to  view  and  monitor  network  operations.  Some  of  the  data  visualization 
tools  need  improvement  for  easier  comprehension. 

All  the  Network  Operations  tools  are  scheduled  for  transition  to  DFC2,  which  will  be 
implemented  in  the  DISA  Network  Centric  Enterprise  Services  (NCES)/Joint  Command  and 
Control  (JC2)  efforts.  The  NetOps  Network  Status  Monitoring  will  transition  to  the  USPACOM 
Theater  C4I  Coordination  Center.  The  Advanced  Intrusion  Detection  Events  (AIDE)  system 
visualizations  (Events  Involving  Critical  Servers  [EICS],  Event  Priority  Chart  [EPC],  and  Link 
Analysis  tools)  will  transition  to  the  DISA  Information  Superiority  Situational  Awareness 
(ISSA)  Portal  initiative. 


So  What? 


Assessments  for  COIs 

The  overall  assessments  of  Suitability,  Usability,  and  Technical  for  each  COI  are  shown  in  the 
following  table.  The  CINC  21  products  that  contribute  to  COI-1  were  generally  considered  to 
help  decision-makers  and  staff  to  process,  digest,  and  assimilate  large  amounts  of  complex 
information.  Most  of  these  products  were  useful  for  the  operational  tasks  and  work  processes, 
and  they  were  relatively  easy  to  use.  Suitability  and  Usability  are  rated  as  Green  for  the 
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combination  of  products  related  to  COI-1.  The  overall  rating  for  COI-1  for  Technical  is  YELLOW 
because  many  of  the  products  were  not  yet  developed  to  the  point  where  they  could  reliably 
provide  dynamic,  real-world  data  to  users.  Overall,  the  Suitability  and  Usability  COI-1 
objectives  are  being  met,  but  there  are  concerns  with  the  DFC2  that  should  be  addressed.  There 
are  still  issues  to  be  resolved  regarding  how  best  to  integrate  all  the  relevant  information  and 
present  it  in  a  useful  manner. 


Critical  Operational  Issue 

Suitability 

Usability 

Technical 

1  -  Decision-Focused 
Visualization 

© 

© 

A 

2  -  Knowledge  Management 

A 

A 

A 

3  -  Collaboration 

© 

A 

A 

4  -  Enterprise  Awareness  & 
Network  Security 

© 

A 

A 

The  CINC  2 1  products  and  concepts  provided  many  exciting  and  innovative  approaches  to 
knowledge  management.  The  current  versions  of  the  products,  however,  need  further  refinement 
to  achieve  these  goals  to  the  levels  needed  by  operational  users.  For  COI-2,  Suitability,  Usability, 
and  Technical  ratings  were  all  YELLOW.  Suitability  can  be  improved  by  linking  more  of  the  key 
data  sources  together.  Usability  can  be  improved  by  making  it  easier  for  operational  users  to 
ingest  and  link  messages  and  other  data  together.  Many  technical  challenges  remain  to  be 
resolved  concerning  how  to  enable  CINC  2 1  products  to  deliver  the  range  of  operational  content 
that  users  need  in  an  accurate  and  reliable  manner. 

As  called  for  in  COI-3,  collaboration  technology  is  an  important  capability  that  can  enhance 
distributed  decision-making  substantially.  Users  consistently  reinforced  this,  reporting  that  the 
CINC  2 1  collaboration  products  were  very  useful  for  their  operational  needs.  Suitability,  was, 
therefore,  rated  as  Green.  Some  of  the  collaboration  products  were  difficult  to  configure  and  to 
use,  however.  This  difficulty  accounts  for  the  YELLOW  rating  for  Usability.  The  biggest  obstacle 
for  collaboration  technology  was  clearly  the  technical  issues;  Technical  was  also  rated  as 
YELLOW.  The  products  were  often  incompatible  with  each  other,  required  excessive  bandwidth, 
and  operated  unreliably. 

COI-4  calls  for  advances  in  the  secure  exchange  of  information  across  multiple,  distributed 
networks,  databases,  and  applications.  Given  successes  with  the  CINC  2 1  products  developed  to 
satisfy  COI-4,  the  Network  Operations  tool  suite  will  be  featured  in  next-generation 
implementations.  Suitability  for  COI-4  was  rated  as  Green.  There  were  some  operational  and 
integration  issues,  however,  with  information  assurance  tools  that  impede  achievement  of  all 
requirements.  Ratings  for  Usability  and  Technical  were  YELLOW. 
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Transition 

Many  varied  types  of  technical  products  were  developed  as  part  of  the  CINC  2 1  program.  While 
some  applications  directly  support  users’  work  processes  and  tasks,  others  involve  system 
architecture  concepts  and  enterprise  infrastructure  that  enable  other  products  to  function 
effectively.  Some  products  explore  the  feasibility  of  potentially  relevant  techniques,  and  other 
interim  products  were  abandoned  as  the  focus  changed  due  to  external  factors,  like  Navy  Marine 
Corps  Internet  (NMCI). 

CINC  2 1  products  that  are  candidates  for  transition  were  identified,  based  on  their  Suitability, 
Usability,  and  Technical  assessments.  These  transition  products  include  the  following  : 

•  TeamApp 

•  Task  Management  System 

•  Master  Calendar 

•  Fused  Battlespace  View 

•  Request  for  Information  (RFI)  Manager 

•  Network  Operational  Tools  and  Services 

•  Visualization  (Displays) 

•  Message  Tracker 

•  DFC2 

•  Collaboration 

Specific  enhancements  are  recommended  that  are  intended  to  help  these  products  achieve  an 
acceptable  level  of  performance  in  an  operational  setting.  Transition  priorities  and  an 
implementation  timeline  were  determined  by  the  amount  of  effort  estimated  to  be  required  to 
make  the  recommended  enhancements. 

The  CINC  21  products  that  have  been  identified  for  transition  need  to  be  incorporated  into  a 
feasible  standard  operating  procedure  (SOP)  that  includes  defined  staff  roles  and  work  processes. 
While  substantial  variations  in  the  details  of  the  SOP  are  expected  across  COCOMs  because  of 
their  unique  responsibilities  and  operating  areas,  there  should  be  a  common  framework.  The 
CINC  2 1  operating  concepts  and  system  architecture  enable  the  products  to  be  used  in  a  common 
manner  that  generalizes  to  many  different  environments,  including  COCOMs  and  Joint  Task 
Forces  (JTFs). 

Various  enhancements  will  need  to  be  made  to  the  CINC  21  products  to  make  them  more 
suitable,  usable,  and  technically  sound  in  operational  settings.  Several  specific  suggestions  for 
these  enhancements  have  been  provided  for  CINC  2 1  products  in  this  report  and  in  previous 
assessment  reports.  Operational  users  should  have  periodic  opportunities  to  review  the  technical 
modifications  and  to  provide  feedback  on  the  enhancements. 

Once  the  product  enhancements  have  been  made  and  reviewed  by  representative  operational 
users,  a  variety  of  supporting  documentation  will  need  to  be  prepared.  This  documentation 
includes  training  materials,  system  support  and  administration  procedures,  and  similar  reference 
materials. 
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Finally,  the  enhanced  product  capabilities  should  be  assessed  in  a  realistic  operational  context. 
This  assessment  is  not  focused  on  determining  military  utility  but  rather  on  verifying  that  the 
product  enhancements  function  properly  and  satisfy  user  needs  and  system  requirements. 
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Introduction 


Background 


Theater  Combatant  Commanders  (COCOMs)  are  actively  engaged  in  world  events  across  the 
spectrum  of  operations.  Operational  crises  often  involve  other  nations,  peer  agencies,  and/or  non¬ 
government  organization  (NGO)  partners.  COCOMs  must  improve  the  functionality  of  these 
complex  relationships.  COCOMs  operate  in  an  information  environment  where  they  and  their 
staffs  are  simultaneously  overwhelmed  with  information  and  left  starved  for  knowledge. 

Once  data  are  available,  humans  manually  cull  through  the  infonnation  to  determine  relevance 
to  a  problem  at  hand.  To  be  used  optimally,  information  must  be  organized  around  mission  areas, 
foundation  and  functional  areas,  tasks,  and  processes.  Information  must  then  be  managed  to  ensure 
consistency  and  shared  with  distributed  sites.  Dispersing  the  information  across  multiple  sources, 
security  levels,  and  networks  increases  the  complexity  of  sharing  and  using  the  infonnation. 

Joint  Vision  2020  (JV2020)1  builds  upon  and  extends  the  conceptual  template  established  by  Joint 
Vision  2010  to  guide  the  transformation  of  America's  Armed  Forces  (Joint  Staff,  2002).  The 
overall  goal  of  the  transformation  is  the  creation  of  a  force  that  is  dominant  across  the  full 
spectrum  of  military  operations — much  of  it  never  experienced  before.  This  world  will  be  an 
information-dependent,  information-saturated  world,  and  a  much  less  predictable  world  than  the 
one  known  before.  Our  advantage  must  come  from  information  superiority  and  innovations  that 
enable  us  to  take  advantage  of  technology  to  achieve  superior  warfighting  effectiveness. 

As  the  military  moves  toward  JV2020  solutions,  COCOMs  are  challenged  by  intricate  information 
and  force  structure  relationships.  Information  superiority  will  be  increasingly  essential  to  success¬ 
ful  operations  in  the  21st  Century.  However,  we  need  more  than  infonnation  superiority;  we  must 
achieve  decision  superiority,  which  is  the  ability  to  effectively  use  and  quickly  exploit  information. 
Forces  and  decision-makers  will  be  distributed,  but  must  function  as  one  cohesive  team. 


1  Joint  Chiefs  of  Staff.  2000.  Joint  Vision  2020.  Department  of  Defense,  Washington,  DC. 
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As  the  military  moves  toward  JV2020  solutions,  COCOMs  are  challenged  by  intricate 
information  and  force  structure  relationships.  Information  superiority  will  be  increasingly 
essential  to  successful  operations  in  the  21st  Century.  However,  we  need  more  than  information 
superiority;  we  must  achieve  decision  superiority,  which  is  the  ability  to  effectively  use  and 
quickly  exploit  information.  Forces  and  decision-makers  will  be  distributed,  but  must  function  as 
one  cohesive  team. 

The  United  States  Pacific  Command  (USPACOM)  has  initiated  the  Theater  Plan  for 
Transformation.  The  Theater  Plan  for  Transfonnation  provides  the  guidance  for  developing  an 
information  infrastructure  to  enhance  mission  effectiveness  and  improve  situational  awareness. 
The  USPACOM  Chief  Information  Officer  has  defined  the  Information  Capabilities  Framework 
(ICF)  for  developing  an  effective  information  management  infrastructure.  Figure  1  illustrates  a 
high-level  view  of  the  ICF  with  one  layer  expanded  to  highlight  how  users  interface  with  the 
information  infrastructure  through  common,  operational,  and  operational  support  applications. 

The  knowledge  presentation  framework  (Figure  2)  provides  the  structure  for  capturing 
information  sets,  rules,  and  situation  awareness  within  contexts  and  making  it  all  available  to 
others.  The  basic  element  for  connecting  together  all  parts  is  the  “view.”  A  view  is  the  product  of 
information  producers  to  provide  to  decision-makers  via  the  network.  In  its  most  complex  form, 
a  view  is  a  template  that  is  connected  to  infonnation  elements  using  Extensible  Markup 
Language  (XML).  In  this  case,  the  view  assembles  infonnation  elements  using  XML  Extensible 
Stylesheet  Language  (XSL).  In  its  most  simple  form,  a  view  can  be  a  simple  page  of  information, 
a  briefing  slide,  a  video  feed,  a  picture,  an  image  or  a  combination  of  all.  A  key  point  is  that 
views  show  “live”  information,  not  simply  static  pictures  or  pages.  Information  or  sensor  feeds 
(including  video)  are  updated  in  the  information  base  and  therefore  are  automatically  updated  in 
the  view  because  they  are  linked  via  XML  tags.  To  create  views,  Knowledge  View  Control  tools 
are  used  to  assemble  infonnation  elements  and  objects  into  templates,  pages,  and  windows.  Once 
a  knowledge  worker  or  subject  matter  expert  creates  a  view,  he  stores  it  into  a  Knowledge  View 
Library.  Any  other  knowledge  worker  or  decision-maker  can  then  find  and  select  views  that  are 
needed  to  insert  into  “cells.”  Multiple  cells  are  then  arranged  in  a  “decision  support  workspace.” 
“Frames”  bound  the  cells  of  a  decision  support  workspace,”  and  these  frames  can  be  moved  to 
resize  and  reposition  the  cells  into  the  desired  configuration.  Decision  support  workspaces  can 
then  be  used  for  status  boards,  large  screen  displays,  and  fixed  windows  to  support  a 
collaboration  session  with  a  combination  of  a  chat  window,  video  window,  common  operational 
picture,  whiteboard,  and  the  like.  Like  views,  decision-support  workspaces  can  be  stored  in  a 
library  for  other  knowledge  workers  and  decision-makers  to  use  as  well.  A  knowledge  core  links 
the  knowledge  views,  decision  support  workspace  tools,  office  automation  tools,  and  standard 
query  and  report  tools  to  the  information  base.  The  knowledge  core  comprises  the  protocols, 
formats,  rules,  discipline  and  framework  needed  to  link  infonnation  with  experts  and  users  to 
knowledge  products.  The  knowledge  presentation  framework  links  external  information  and 
knowledge  sources  with  a  command’s  internal  information  sets.  As  improvements  in  the 
information  technology  migrate  into  the  Department  of  Defense,  USPACOM,  and  other 
Combatant  Commanders  can  readily  adapt  their  organizations  to  provide  more  effective 
command  and  control  planning  and  decision-making. 
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Figure  1.  ICF  high-level  organization. 


Figure  2.  Knowledge  presentation  framework. 
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Goal  of  the  CINC  21  Advanced  Concept  Technology 
Demonstration 


The  Commander  in  Chief  21st  Century  (CINC  21)  is  an  Advanced  Concept  Technology 
Demonstration  (ACTD)  that  was  designed  to  provide  broad  solutions  to  these  problems.  CINC 
21’s  goal  is  to  improve  command  and  control  of  forces  by  exploiting  advanced  visualization 
techniques  and  decision  support  systems  (e.g.,  presentations,  cueing,  triggers,  alerts), 
collaboration  capabilities,  and  knowledge  and  enterprise  management  technologies.  CINC  21 
used  information  technology  to  reduce  dependence  on  centralized,  single-crisis  command  centers 
in  favor  of  decentralized,  rapidly  configured  coalition  and  interagency  enclaves. 


Purpose  of  the  Joint  Military  Utility  Assessment 


To  evaluate  the  technologies  developed  as  part  of  the  CINC  21  ACTD,  a  Joint  Military  Utility 
Assessment  (JMUA)  was  planned  and  executed.  The  purpose  of  the  JMUA  was  to  determine 
how  well  the  CINC  2 1  products  met  warfighter  requirements.  The  general  approach  for  the 
JMUA  involved  several  phases — some  analytic  and  others  empirical.  In  this  way,  the  most 
complete  technology  assessment  would  be  conducted  in  an  efficient  manner  that  is  minimally 
intrusive  on  any  exercises  or  other  events  in  which  CINC  21  was  being  demonstrated/evaluated. 


Organization  of  this  Report 


This  report  is  organized  according  to  the  following  sections: 

•  WHAT  IS  CINC  21? 

o  an  overview  of  the  ACTD  and  the  products 

•  WHY  DEVELOP  CINC  21? 

o  a  description  of  the  operational  issues  and  requirements 

•  WHO  HAD  ASSESSMENT  RESPONSIBILITY? 

o  the  roles  and  responsibilities  of  the  assessment  team 

•  HOW  WAS  CINC  21  ASSESSED? 

o  the  approach  and  assessment  criteria  used  for  the  JMUA 

•  OUTCOME 

o  the  detailed  analysis  of  the  results 

•  SO  WHAT? 

o  a  list  of  conclusions,  recommendations,  and  a  plan  for  transition 
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What  is  CINC  21? 


Overview 


During  crises,  the  flood  of  data  that  becomes  available  is  such  that  accessing  information, 
placing  it  in  context,  and  understanding  its  relevance  is  extremely  difficult.  When  faced  with 
multiple  crises  in  the  theater,  the  deluge  of  information  is  even  greater.  Today's  battle  staffs  are 
constrained  by  limited  command  center  spaces,  lack  of  intelligent  display  systems,  and 
organizationally  limited  information  flow.  To  complicate  the  problem,  networks  that  are  located 
away  from  major  headquarters  elements  are  severely  restricted  in  capacity.  CINC  21’s  goal  was 
to  improve  command  and  control  of  forces  by  exploiting  advanced  visualization  techniques  and 
decision  support  systems  (e.g.,  presentations,  cueing,  triggers,  alerts),  collaboration  capabilities, 
and  knowledge  and  enterprise  management  technologies.  CINC  21  used  information  technology 
to  reduce  dependence  on  centralized,  single-crisis  command  centers  in  favor  of  decentralized, 
rapidly  configured  coalition  and  interagency  enclaves. 

CINC  2 1  focused  on  accelerating  the  ability  of  decision-makers  to  understand  the  impact  of 
events  and  to  improve  their  ability  to  collaborate,  plan,  and  make  decisions  on  appropriate 
courses  of  action  in  situations  involving  multiple  concurrent  theater  operations  and  coalition 
activities.  The  ACTD  is  a  5-year  effort  that  continues  through  Fiscal  Year  (FY)  2004.  CINC  21 
also  addressed  the  problems  associated  with  the  need  to  collaborate,  plan,  and  decide  with  all 
essential  parties  regarding  appropriate  courses  of  action  in  situations  involving  multiple, 
concurrent  theater  operations  and  coalition  activities. 

The  major  objectives  of  CINC  21  are  as  follows: 

1 .  Improve  situational  awareness  and  understanding  by  means  of  (a)  shared  understanding 
of  operational  situation,  (b)  ability  to  scale  and  tailor  visualization,  and 

(c)  advanced  decision  support  and  knowledge  management  tools. 

2.  Demonstrate  and  synchronize  distributed  decision-making,  collaboration,  and 
information  management/information  dissemination  tools  among  joint,  coalition,  inter¬ 
agency,  and  non-governmental  organization  (NGO)  partners. 

3.  Enable  command  of  the  infonnation  enterprise  via  advanced  enterprise  management  tools 
and  user-specified  and  prioritized  operational  products. 
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Products 


The  vision  of  CINC  21  is  to  develop  and  assess  a  decision-focused  command  and  control 
framework  that  enables  joint  commands  to  dynamically  manage  and  focus  organization, 
processes,  resources,  and  information  to  improve  situational  awareness  and  decision-making. 
CINC  21  comprises: 

•  A  Set  of  Concepts 

•  An  Enterprise  Command  and  Control  Infrastructure 

•  Decision-Focused  Command  and  Control  Applications 

•  Operational  Packages 

•  Displays 

•  A  Collaboration  Capability 

•  Network  Operations  Tools  and  Services 

•  Special  Projects 

Concepts 

The  advanced  concept  is  intellectual  property  associated  with  a  change  in  business  process.  It 
provides  shared  views  across  the  enterprise  to  facilitate  a  common  understanding  of  the  situation 
and  the  decision  environment.  The  views  allow  the  command  to  dynamically  manage  and  focus 
across  the  immediate  command  and  desired  supporting  commands.  The  functional  capability  is  a 
shared  set  of  views  across  the  enterprise  to  facilitate  a  common  understanding  of  the  situation 
and  the  decision  environment.  The  views  allow  the  command  to  dynamically  manage  decisions 
and  focus  across  the  immediate  command  and  desired  supporting  commands.  This  common 
understanding  can  be  achieved  in  a  command  center,  in  an  office,  and  even  on  a  laptop  in  transit 
simultaneously.  CINC  21  defined  four  concepts. 

Enterprise  Architecture 

The  Enterprise  Architecture  is  the  underlying  assembly  of  technologies  of  hardware  and  software 
components  necessary  to  support  the  battle  rhythm  concept.  These  technologies  provide  a  three- 
tier  architecture  that  cleanly  separates  data  from  business  logic  from  presentation.  It  provides 
knowledge  management  based  on  metadata  (XML).  It  highlights  the  use  of  web  services  and 
databases  to  provide  heterogeneous  information  access  across  the  enterprise.  This  access  allows 
dynamic  content  sharing  and  dynamic  views  to  create  knowledge  and  support  decision-making 
using  various  display  media. 

Optimal  Knowledge  Wall  Usage 

The  Knowledge  Wall  is  a  high-resolution  group  display  that  is  intended  to  promote  shared 
visualization  and  shared  situation  awareness  among  co-located,  functionally  distinct  teams.  The 
optimal  usage  of  this  group  display  technology  is  a  concept  that  involves  business  processes, 
display  layout,  and  decision  space  management  for  configuring  large  volumes  of  information  in 
useful  and  usable  ways.  An  optimally  configured  knowledge  wall  promotes  collaboration  and 
information  exchange  among  team  members  and  facilitates  the  function-specific  work  perfonned 
by  individual  team  members.  Lessons  learned  via  iterative  developments  at  USSTRATCOM 
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provide  important  guidance  toward  optimal  knowledge  wall  usage,  although  further 
enhancements  in  this  concept  are  anticipated  as  additional  technical  capabilities  and  products 
become  available. 


Decision-Focused  Command  and  Control 

Decision-Focused  Command  and  Control  (DFC2)  is  a  concept  that  enables  Joint  Commands  to 
dynamically  manage  and  focus  organization,  processes,  resources,  and  information  across  the 
enterprise  to  improve  Situational  Awareness  and  Decision  Making.  Synchronized  “battle 
rhythm”  is  the  mechanism  that  provides  the  warfighters  across  the  enterprise  with  common 
views  and  goals  in  near  real  time. 

CINC  21  Concept  of  Operations  (CONOPS) 

The  CINC  21  C2  Concept  of  Operations  (CONOPS)  describes  the  information  model  across  the 
enterprise.  It  includes  the  warfighter  processes  (types  of  common  activities)  and  the  information 
activities  performed  within  each  warfighter  process.  It  includes  a  data  model,  process  flows,  and 
information  flows/activities  at  nodes  in  the  flow.  If  possible,  timelines  are  useful. 

Enterprise  Command  and  Control  Infrastructure 

An  enterprise  command  and  control  infrastructure  was  developed  as  part  of  CINC  2 1 .  The 
infrastructure  consists  of  an  aggregation  of  commercial  and  government  off-the-shelf  products 
running  in  an  open  standards  environment.  This  infrastructure  provided  information-level 
services.  These  services  allowed  access,  manipulation,  display,  and  collaboration  at  the 
information  level.  The  infrastructure  components  consist  of  the  following  components: 

Enterprise  Workstation 

Enterprise  Workstation  is  a  common  client  workstation  for  USPACOM.  It  runs  CINC  21  client 
and  USPACOM  client  software  on  hardware  with  appropriate  computational  power  and  memory. 

Enterprise  Oracle®  Database 

Enterprise  Oracle®  Database  is  commercial  software  (8i  or  9i)  that  provides  persistence  for  the 
knowledge  core  in  the  infrastructure. 

Enterprise  Application  Server 

Enterprise  Application  Server  was  implemented  via  WEBLOGIC  6.1.  It  is  the  tool  to  provide 
interoperability  of  information  through  conversion  to  HTML  format  Enterprise  Portal  Server. 

Enterprise  Portal  Server 

Enterprise  Portal  Server  was  implemented  via  Digital  Dashboard.  Its  purpose  was  to  provide 
dynamically  configured  views  of  infonnation  layout. 

Business  Process  Management  Server 

Business  Process  Management  Server  was  not  implemented  but  is  expected  to  become  part  of 
CINC  21  during  the  transition  phase  (FY  2003  through  2004). 
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Knowledge  Core  (XML  Data  Services) 

Knowledge  Core  (XML  Data  Services).  The  knowledge  core  is  a  collection  of  software  servers 
that  provide  infonnation  “tagging”  and  persistence.  The  core  provides  tools  for  XML  generation 
for  geo-spatial  products,  web  pages,  and  documents  as  well  as  indexing  and  retrieval  of 
persistent  information.  It  can  operate  locally  or  federated  using  XML  parameters  as  pointers. 

Decision-Focused  Command  and  Control  Applications 

Decision-Focused  Command  and  Control  (DFC2)  is  implemented  as  a  set  of  software 
components  that  facilitate  management  of  organization,  information,  process  and  resource 
functions.  The  software  components  are  organized  in  three  layers  (tiers):  presentation,  business 
process  and  data.  These  components  provide  information  and  knowledge  navigation,  status, 
product  linking,  decision  summaries,  and  access  control  across  the  enterprise,  along  with  a  data 
schema  (based  on  data  model)  that  relates  these  components  to  the  information  in  the  data  layer. 

Presentation  Layer 

This  layer  is  the  interface  to  the  warfighter.  It  provides  the  visualization  of  the  underlying 
information  to  provide  knowledge. 

Decision  Space  Navigation  Schema.  This  schema  is  a  specification  of  the  elements  required  to 
support  the  warfighters’  visualization  requirements. 

Presentation  Viewers.  The  hierarchical  viewer  is  implemented  as  a  tree  for  navigation  and 
tables  for  information  presentation.  The  battle  rhythm  viewer  is  implemented  as  a  tree  for 
navigation  and  a  timeline  for  information  presentation. 

Shared  Visualizations.  These  visualizations  are  layouts  and  web  parts  that  provide  access  to  the 
underlying  knowledge  core/objects.  They  provide  a  means  to  ask  for  information  to  be  displayed, 
and  they  provide  a  way  to  actually  present  the  information  to  the  viewers  from  the  common 
information  in  the  knowledge  core,  including  portal  layouts,  fonts,  and  colors. 

Portal  Configurations.  These  configurations  are  typical  layouts  that  support  the  shared 
visualizations,  including  decision  space,  user  management,  and  infonnation  management. 

Business  Layer 

This  layer  is  the  logic  (implemented  as  Enterprise  Java  Beans)  that  represents  the  command’s 
business  model.  It  manipulates  the  data/information  to  provide  operational  value  to  the 
warfighter. 

Business  Logic  Utilities.  These  utilities  are  tools  that  provide  capabilities 
such  as  time/date,  filtering,  and  status  rules. 

Decision  Space  Management.  These  tools  allow  the  creation,  editing,  copying,  pasting,  and 
deleting  of  business  objects  (operation,  decision  points,  etc.).  These  objects  can  then  be 
configured  as  hierarchical  lists  or  timelines  to  navigate  the  decision  space. 

Information  Management.  These  tools  allow  information  product  linking.  These  tools  also 
track  changes  and  provide  alerts. 
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Business  Process  Management.  These  tools  are  for  workspace  management  and  workflow. 

System  Administration.  These  tools  provide  general  user  authentication  with  permissions  and 
selected  access. 


Data  Layer 

This  layer  provides  the  actual  interaction  with  data  and  information.  The  interaction  is  at  the 
content  level  (XML  Services) 

Decision  Space  Data  Schema.  This  schema  is  a  specification  of  the  data  required  to  support  the 
business  process.  It  is  the  basis  for  the  XML  generated  and  accessed  in  the  knowledge  core. 

Access  Control  Management.  These  tools  provide  the  mechanisms  for  the  System 
Administration  to  be  executed  at  the  data  layer. 

Pedigree  Management.  This  tool  provides  tracking  of  persons  making  changes  and  times. 
Schema  Management.  These  tools  facilitate  changes  in  schema. 

Operational  Packages 

The  Operational  Packages  are  applications  that  use  the  underlying  enterprise  architecture  to 
provide  new  production  capabilities  for  echelons  below  the  decision-maker.  These  applications 
result  in  information  stored  in  databases  and  available  for  supporting  content  that  can  be  aggre¬ 
gated/rolled  up  for  the  decision-maker.  They  upgrade  existing  desktop-oriented  applications  or 
provide  new  capabilities  to  improve  the  Action-Officer-level  tasks.  These  upgrades  and  new 
capability  then  provided  metadata  and  persistence  for  information  reuse.  The  following  sections 
describe  the  operational  packages. 

Common  Operational  Picture/Geo-Spatial  Visualization 

Area  of  Responsibility  (AOR)  Basing 

This  application  uses  a  GOTS  decision  aid  (XIS  ViewPoint)  to  manipulate  and  display 
information  in  a  geo-spatial  context.  The  application  ingests  three  static  databases  (one  is  a 
browser-based  access  to  the  database).  The  application  also  ingests  two  dynamic  sources.  One 
source  is  a  transportation-related  web  page  where  the  information  updates  are  pulled  from  the 
page  and  stored  in  a  database,  which  is  then  ingested  and  manipulated  as  XML  in  the  CINC  2 1 
Knowledge  Core.  The  second  dynamic  source  is  the  Theater  Logistics  Operations  Center  hot 
spot.  All  the  infonnation  ingested  can  be  manipulated,  overlaid,  and  interrogated  for  related 
information.  All  displays  can  be  saved  to  the  Knowledge  Core  for  reuse  and/or  converted 
directly  to  PowerPoint. 

Fused  Battlespace  View  (FBV) 

This  tool  is  a  joint  USSTRATCOM/CINC  21  application  that  accesses  and  manipulates 
USSTRATCOM  information.  It  is  a  JAVA  program  that  accesses  the  USSTRATCOM  Know¬ 
ledge  Core  and  displays  their  strategic  information  in  a  fixed  structure.  It  provides  a  geo-spatial 
context  with  drill-down  capability.  It  is  tightly  integrated  to  the  USSTRATCOM  operation 
process. 


9 


Joint  Military  Utility  Assessment 


Briefing/Information  Summary 

Consequence  Management  Automated  Brief 

This  brief  is  a  CINC  21 -developed  application.  Its  purpose  is  to  get  briefing  information  in  a 
database  where  it  can  be  reused  and  presented  as  PowerPoint,  web  pages,  and  directly  as  content 
to  near  real-time  summaries.  It  provides  a  web  fonn  for  data  entry  that  is  stored  in  the  knowledge 
core.  It  can  latter  be  manipulated  as  XML  content,  if  desired.  The  actual  capability  is  currently 
limited  with  some  fixed  formats  as  described  by  USPACOM  users. 

Course  of  Action  (CO A)  Matrix 

This  matrix  CINC  21  application  was  developed  for  USSTRATCOM  and  provides  a  data  model 
based  on  the  DFC2  schema  that  is  implemented  as  a  CINC  21  table  with  knowledge  core  drill¬ 
down.  It  provides  a  mechanism  for  aggregating  and  displaying  top-level  decision  information 
and  execution  information.  It  also  provides  a  mechanism  for  organized  drill-down. 

Status  Tracking  and  Linking 

Message  Tracker 

This  tool  is  a  CINC  21 -developed  application.  It  allows  operators  to  read  messages  and  link  them 
to  either  an  Operation  or  a  Commander's  Critical  Information  Requirements  (CCIRs).  This  tool 
basically  puts  metadata  to  the  messages  and  allows  them  to  be  imported  to  DFC2  based  on  the 
metadata.  It  is  intended  as  a  production  tool  that  gets  information  to  the  knowledge  core  and 
allows  reuse.  The  application  is  implemented  as  a  JAVA  server  page. 

Request  for  Information  ( RF1 )  Manager 

Currently,  this  tool  is  a  CINC  2 1  stand-alone  application.  It  is  not  linked  to  the  knowledge  core 
at  this  time.  At  USPACOM  headquarters,  it  uses  local  databases  and  web  forms  that  link  to  the 
local  database.  This  combination  allows  for  a  local  capability  to  track  requests  and  identify  due 
dates,  status,  and  responsible  individual/organization.  The  tool  has  been  used  extensively  in  the 
venue  for  which  it  was  originally  conceived — at  the  Joint  Task  Force  headquarters.  It  has  been 
used  operationally  by  III  MEF  and  COMSEVENTHFLT.  MARFORPAC  and  US  Forces  Japan 
are  currently  evaluating  it  for  the  RFI  management  software  for  their  commands. 

TeamApp 

TeamApp  is  a  web-based  tool  that  provides  a  template  for  structuring  and  managing  event 
information  resources.  The  main  feature  provides  participants  (users  or  guests)  access  to  all  the 
information  that  is  pertinent  to  a  given  event.  Information  is  easily  updated  and  sorted. 


Time-Based  Event  Management 

Task  Management  System  (TMS) 

The  Task  management  System  (TMS)  provides  a  web-based  structure  for  tracking  various 
events.  A  task  can  be  a  request  from  an  office  to  its  subordinates  to  perform  certain  actions.  A 
task  can  also  be  an  item  sent  for  a  lower-level  office  to  its  parent  offices  for  review  and  approval. 
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The  TMS  helps  users  to  (1)  track  tasks  from  start  to  finish;  (2)  provide  a  single  location  for 
instructions,  responses,  documents,  and  e-mails  for  a  task;  and  (3)  archive  task  information  for 
easier  searching.  It  enables  an  organization  to  keep  track  of  who  is  assigned  to  tasks,  when  tasks 
are  due,  and  what  the  task  is  about. 


Master  Calendar 

The  USPACOM  Master  Calendar  provides  an  integrated  view  of  significant  events  relevant  to 
the  Commander  and  the  headquarters  staff.  The  display  is  used  for  event  de-confliction  and 
general  situational  awareness  of  important  events  affecting  the  command.  The  application  is  a 
web-based  tool,  resides  on  the  headquarters  Intranet  system  and  operates  within  the  TeamApp 
environment.  The  interface  allows  users  to  choose  and  toggle  between  multiple  event  displays, 
including  a  standard  monthly  event  calendar  view,  and  Gantt  views  that  show  how  events 
overlap  over  extended  periods  of  time.  These  displays  can  be  customized  and  filtered  to  display 
only  event  information  relevant  to  a  particular  user. 

JWID  Coalition  Information  Interoperability  Network  (COINS) 

The  Coalition  Information  Interoperablity  Network  (COINS)  is  an  operational  package  that 
provides  interoperability  of  coalition  (U.S.,  UK,  CA,  AU)  infonnation  at  the  information  content 
level.  It  was  used  in  JWID  02.  It  uses  the  Enterprise  C2  Infrastructure  with  special  focus  on  the 
knowledge  core  and  XML  services. 

Composable  Frames 

Composable  Frames,  an  early  example  of  web-based  technology,  was  only  used  in  preliminary 
demonstrations  and  only  assessed  informally. 

Displays 

Group  Displays:  Large  Screen  (Standard  Resolution) 

The  USPACOM  video  wall  is  a  3  x  9  matrix  of  2 1 -inch  video  monitors  that  are  affixed  to  a  wall 
and  used  to  display  desktop  views. 

Group  Displays:  Large  Screen  (High  Resolution) 

Six  4  foot  x  5  foot  cubes  are  arranged  in  a  2  x  3  matrix.  The  total  size  of  the  display  is  8  feet  x  15 
feet  with  a  1280  x  1024  cube-to-cube  resolution.  Activu  Control  Manager  and  Gateway  and 
Activu  Display  software  run  on  two  servers  to  drive  the  matrix  of  the  display  cubes. 

Workstation  Displays:  Multiple  Side-by-Side  Flat  Monitors 

These  monitors  are  commercial  flat  panel  displays  that  are  relatively  inexpensive.  They  require 
an  appropriate  video  driver  dictated  by  the  client  operating  system  and  the  display  capability. 
They  can  be  two  or  three-panel  (or  more)  and  provide  a  space  for  transforming  information  to 
knowledge  by  the  user. 

Workstation  Displays:  PV290  Multi-Screen  Desktop  Display 

The  Panoram  PV290  DSK  is  a  three-screen  Liquid  Crystal  Display  (LCD)  monitor  that  is 
intended  for  use  in  various  applied  settings.  Its  major  advantage  relative  to  three  conventional 
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monitors  is  that  the  three  screens  of  the  PV290  DSK  are  separated  by  a  0.625-inch  wide  bezel. 
This  narrow  boundary  enables  the  PV290  DSK  to  present  a  more  seamless  image  than 
conventional  monitors,  which  typically  have  a  1.5  to  2-inch  wide  housing  border,  creating  a 
3  to  4-inch  wide  border  between  adjacent  screens.  The  overall  image  size  across  its  three  equal¬ 
sized  screens  (18.1  inches  diagonal)  is  43.5  inches  x  1 1.5  inches,  making  it  approximately 
equivalent  to  three  19-inch  monitors.  The  resolution  of  each  screen  is  1280  x  1024,  resulting  in 
an  overall  resolution  of  3840  x  1024. 


Collaboration 

The  collaboration  services  are  a  collection  of  software  servers  and  products  that  support  the 
Defense  Collaboration  Tool  Suite  (DCTS).  There  are  Portal  Enabled  Extensions  to  DCTS  that 
allow  collaborations  to  run  in  portal  environments.  APIs  for  Embedded  Collaboration  is  software 
that  links  collaboration  spaces  to  DFC2  objects.  Information  Work  Spaces  (IWS)  were  also  made 
available  to  groups  of  operators. 

Network  Operations  Tools  and  Services 

The  network  operations  aggregate  commercial  applications  information  through  the  CINC  2 1 
knowledge  core.  The  applications  are  ingested  in  the  knowledge  core  with  their  XML  tags.  This 
information  is  then  aggregated  and  displayed  as  tables  (in  near  real  time)  via  business  rules.  The 
presentation  is  a  matrix  with  drill-down  capability  that  presents  the  state  of  operational  servers 
and  networks.  Network  operations  were  supported  by  monitoring  and  infonnation  assurance 
tools. 


Theater  C4ISR  Status  Grid 

This  grid  is  a  graphic  display,  broken  down  by  theater  organization  and  type  of  resource.  It 
displays  a  rollup  of  the  Remedy  Trouble  Ticket  Workflow  tickets  assigned  to  each  location  and 
resource.  A  drill-down  capability  allows  the  operator  to  see  the  actual  ticket. 

Network  Operations  Status  Monitoring 

Remedy  Trouble  Ticket  Workflow 

The  Remedy  Trouble  Ticket  Workflow  system  runs  the  Network  Monitoring  Tools.  It  processes 
open  tickets  (network  problems)  into  an  XML  formatted  file  that  represents  the  status  of 
resources  located  within  the  theater  organization.  The  representation  drives  the  Command, 
Control,  Communications,  Computers,  Intelligence,  and  Reconnaissance  (C4IR)  Status  Grid 
display  and  integrate  with  other  systems. 

Events  Involving  Critical  Servers  (EICS) 

The  Events  Involving  Critical  Servers  (EICS)  monitor  is  an  information  assurance  monitor  that 
allows  an  operator  to  describe  a  set  of  critical  servers  in  terms  of  easily  understood  aliases.  It 
shows  the  total  number  of  times  the  server  has  been  an  attacker  or  has  been  attacked. 


Event  Priority  Chart  (EPC) 

The  Event  Priority  Chart  (EPC)  is  an  information  assurance  tool  that  shows  Automated  Intrusion 
Detection  Environment  (AIDE)  system  event  activity  over  the  last  30  days. 
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Link  Analysis  Tool 

The  Link  Analysis  Tool  is  an  information  assurance  tool  that  displays  three  categories  of  nodes: 
external,  internal,  and  critical.  Links  are  drawn  between  nodes  to  indicate  attacker/attackee 
relationships. 

One-Way  Proxy 

The  One-Way  Proxy  is  an  information  assurance  tool  that  allows  secure  transport  of  data  from  a 
low  security  network  to  a  high  security  network. 

Battle  Management  Center  (BMC)  Patrol 

Battle  management  Center  (BMC)  Patrol  is  a  commercial  product.  It  is  an  automated  discovery 
process  that  understands  not  only  when  major  failures  occur,  but  also  interprets  the  accumulation 
of  minor  events  that  lead  to  customer-service-level  degradation.  The  hardware,  software,  and 
processes  define  customer-  centered,  service-level  management. 

Network  Operations  Capability  Brief 

The  Network  Operations  Capability  Brief  describes  the  network  capabilities  of  the  entire  theater. 
It  was  developed  to  be  dynamically  updated,  based  on  what  is  in  the  Remedy  Trouble  Ticket 
Workflow  system  and  what  has  been  entered  as  the  mission  impact  statements. 

Special  Projects 

Special  projects  were  single  attempts  to  develop  a  specialized  tool  or  application.  In  some  cases, 
they  were  hardware  or  software  applications  that  supported  the  enterprise  infrastructure  for  a 
particular  phase  of  the  ACTD,  but  were  found  to  be  unnecessary  or  replaced  as  the  ACTD 
concepts  matured.  The  following  sections  describe  these  special  projects. 

USSTRATCOM  Portal 

The  USSTRATCOM  Portal  is  a  USSTRATCOM/CINC  21  developed  visualization.  It  is 
integrated  to  the  USSTRATCOM  knowledge  core  and  allows  interoperability  with  the  FBV 
through  the  knowledge  core.  It  presents  the  command’s  information  via  a  series  of  portal  views. 

JWID  Administrative  Portal 

This  was  a  specialized  developed  visualization  to  support  the  2002  Joint  Warfighting 
Interoperability  Demonstration  (JWID).  It  was  an  initial  instantiation  of  portal  technology.  It  was 
designed  to  support  the  daily  events  of  the  JWID  exercise.  It  was  superseded  by  more  advanced 
portal  technology. 

Time-Step  Virtual  Private  Network  (VPN) 

Virtual  Private  Networks  (VPNs)  were  used  to  provide  encapsulation  and  encryption  to  protect 
against  external  tampering  and  eavesdropping.  TimeStep  VPN  Gateways  provided  DCTS 
collaboration  capabilities  for  USSTRATCOM  users  interfacing  with  USPACOM  and  Advanced 
Information  Technology  Services  (AITS)-Joint  Program  Office  (JPO)  users.  The  VPN  gateway 
at  the  AITS-JPO  was  configured  to  support  split  tunneling,  with  traffic  to/from  USSTRATCOM 
IP  addresses  getting  encrypted/unencrypted,  while  all  other  traffic  was  configured  to  pass 
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through  the  VPN  Gateway  in  “clear”  unencrypted  mode.  This  allowed  users  in  USPACOM 
without  VPN  capability  to  connect  to  a  collaboration  server  at  the  AITS-JPO  that  was  linked 
with  USSTRATCOM,  supporting  collaboration  sessions  with  all  sites. 

Quality  of  Service  (QoS) 

Quality  of  Service  (QoS)  is  a  set  of  capabilities  that  allow  creation  of  differentiated  services  for 
network  traffic,  thereby  providing  better  service  for  selected  network  traffic.  With  QoS,  you  can 
increase  bandwidth  for  critical  traffic,  limit  bandwidth  for  non-critical  traffic,  and  provide 
consistent  network  response.  Implementing  QoS  can  allow  the  efficient  use  of  expensive 
network  connections. 


Remote  Routing  Access  Server  (RRAS)  VPNs  with  Public  Key  Infrastructure 
(PKI) 

CINC  2 1  fielded  the  Microsoft®  Remote  Routing  Access  Server  (RRAS)  to  provide  better  VPN 

(S) 

client  support.  The  RRAS  server  worked  with  the  VPN  client  that  comes  with  Windows  2000, 
strict  lockdown  procedures  were  performed  on  the  RRAS  server,  and  users  were  given  Public 
Key  Infrastructure  (PKI)  certificates  for  user  authentication. 

VOIP  Suite 

The  Cisco  ®  Voice  Over  Internet  Protocol  (VOIP)  Call  Manager  software  and  Internet  Protocol 
phones  were  provided. 

Internet  Protocol,  Version  6  (IPv6) 

Using  the  IPv6  standard,  a  native  test  bed  was  built  to  provide  communications  over  the 
Unclassified  Defense  Information  Systems  Network  (DISN)-Leading  Edge  Service  (LES) 
network. 


NETIQ 

NetlQ  is  a  network  performance-monitoring  tool.  It  was  provided  for  the  JWID  ’02  exercise.  It 
measures  server  performance  and  network  bandwidth  availability. 

NET  VCR 

The  Network  VCR  provides  detailed  performance  measurements  and  playback  capability  of 
traffic  to  critical  CINC  2 1  servers. 


Multi-Router  Traffic  Graphing  (MRTG)  Tool 

The  Multi-Router  Traffic  Graphing  (MRTG)  tool  was  provided  to  collect  and  graph  aggregate 
traffic  on  router  and  switch  ports. 

3Com®  Embedded  Firewall  (EFW)  Network  Interface  Cards  (NIC) 

The  3Com  Embedded  Firewall  (EFW)  technology  was  provided  to  prevent  insider  attacks  and 
outside  hacks  by  placing  a  low-cost,  easy-to-use  network  access  control  point  directly  in  front  of 
each  critical  host  on  the  network.  The  EFW  network  access  control  point  is  independent  of  the 
host  and  controls  all  accesses  to  and  from  the  host.  It  protects  the  network  from  attacks  launched 
from  the  host  itself,  as  well  as  protecting  the  host  from  network  attacks.  It  also  means  a  user  or 
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malicious  code  that  gains  access  to  administrator  mode  on  the  host  cannot  modify  the 
configuration  of  the  access  control  point. 

Tactical  Logistics  Operations  Center  (TLOC)ZJoint  Military  Command 
(JMC)Support 

Tactical  Logistics  Operations  Center  (TLOC)/Joint  Military  Command  (JMC)  is  CINC  21 
technical  support  work  that  included  the  composable  frames  operational  package  and  day-to-day 
implementation  and  improvements  on-site  to  accommodate  JMC/TLOC  users.  It  also  included 
daily  web  page  data  “scraping”  and  manual  data  evaluation  (vetting). 

Navy  Marine  Corps  Internet  (NMCI)  Certification/Cut  Over 

Navy  Marine  Corps  Internet  (NMCI)  Certification/Cutover  supported  the  NMCI  transition.  It 
included  software  information  and  testing. 


Products  and  the  ICF 


Table  1  provides  a  summarization  of  CINC  21  technologies  as  they  relate  to  the  ICF  components 
depicted  in  Figure  1.  Table  1  also  indicates  which  of  the  CINC  21  technologies  were  directly 
evaluated  as  part  of  the  Joint  Military  Utility  Assessment. 
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Table  1.  ICF  components  and  CINC  21  technologies. 


ICF 

Categories 

ICF 

Components 

CINC  21  Technologies 

Direct 

Assessment? 

Process  and 

Process 

Support 

Manpower 

Components 

Training 

Provisioning 

Facilities 

Enterprise  Workstation . 

No 

Group  Displays  (Std  Res  Large  Screen) . 

Yes 

Group  Displays  (High  Res  Large  Screen)... 

Yes 

Workstation  Displays  (Flat  Panels) . 

Yes 

Workstation  Displays  (PV290) . 

Yes 

Infrastructure 

Collaboration 

Collaboration  Services . 

Yes 

and  System 

Applications 

DFC2  Presentation  Layer . 

Yes 

Components 

Enterprise  Oracle  Database . 

No 

Enterprise  Application  Server . 

No 

Enterprise  Portal  Server . 

No 

Business  Process  Management  Server . 

No 

Area  of  Responsibility  Basing . 

Yes 

Fused  Battlespace  View . 

Yes 

Consequence  Management  Automated 

Yes 

Brief . 

Yes 

Message  Tracker . 

Yes 

Request  for  Information  Manager . 

Yes 

Course  of  Action  Matrix . 

Yes 

Master  Calendar . 

Yes 

Task  Management  System . 

No 

Composable  Frames . 

Yes 

JWID  COINS . 

No 

TLOC/JMC  Support . 

Yes 

STRATCOM  Portal . 

Yes 

JWID  Administrative  Portal . 

Info  Management 

DFC2  Data  Layer . 

No 

Knowledge  Core . 

No 

TeamApp . 

Yes 

Network  Info 

DFC2  Business  Layer . 

No 

Distribution 

Quality  of  Service . 

Yes 

IPv6  . 

Yes 

NetlQ . 

Yes 

NET  VCR . 

Yes 

MRTG . 

Yes 
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Table  1.  ICF  components  and  CINC  21  technologies,  (continued) 


ICF 

Categories 

ICF 

Components 

CINC  21  Technologies 

Direct 

Assessment? 

Infrastructure 

Info  Assurance 

BMC  Patrol . 

Yes 

and  System 

Components 

(continued) 

Theater  C4ISR  Status  Grid . 

Yes 

Network  Ops  Capability  Brief . 

No 

Network  Operations  Status  Monitoring: . 

Remedy  T rouble  Ticket  Workflow 

Yes 

Events  Involving  Critical  Servers 

Event  Priority  Chart 

Link  Analysis  Tool 

One-way  Proxy 

VOIP  Suite . 

No 

VPN . 

Yes 

Time-Step . 

RRAS  VPNs  with  PKI . 

3COM®  EWF  NIC  Cards . 

Yes 

NMCI  Certification/Cut-Over . 

No 

Network 

T  ransport 

Planning 

Components 

Planning  Policy 
Architecture 

Enterprise  Architecture . 

No 

Optimal  Knowledge  Wall  Usaqe . 

No 

Decision-Focused  Command  &  Control . 

No 

CINC  21  Concept  of  Operations . 

No 
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Why  Develop  CINC21? 


Operational  Concept 


The  operational  concept  for  CINC  2 1  is  a  knowledge-enabled  information  sphere  with  tools  and 
applications  to  (1)  improve  situation  awareness  and  understanding,  (2)  facilitate  the  ability  to 
collaborate  when  necessary,  and  (3)  manage  the  information  enterprise  while  transforming  and 
accelerating  the  decision  processes  that  underlay  the  management  of  crisis-contingency 
operations,  theater  7  in  improved  processes,  thereby  increasing  overall  headquarters  readiness, 
efficiency,  and  mission  effectiveness. 


The  four  Critical  Operational  Issues  (COIs)  were: 

•  COI-1 :  Can  advanced  visualization  technology  empower  individuals  to  process,  digest, 
and  assimilate  large  volumes  of  information,  enabling  faster,  more  effective  decisions? 

•  COI-2:  Can  knowledge  management  technology  integrate  information,  context,  and  rules 
to  increase  understanding  and  improve  decision-making? 

•  COI-3:  Can  collaboration  tools  be  employed  to  overcome  tyrannies  of  time,  distance,  and 
system  disparity? 

•  COI-4:  Can  the  collection  of  networks,  databases,  and  applications  be  enhanced  to 
optimize  the  flow  of  information,  with  security  assurance,  across  multiple  network 
enclaves? 


Requirements 


The  COIs  reflect  the  objectives  of  the  ACTD  and  are  supported  by  44  detailed  activity 
requirements.  These  requirements  were  prioritized  and  grouped  according  to  the  four  COIs  as 
displayed  in  Table  2.  Although  the  CINC  21  technology  products  that  were  developed  focused 
on  the  top  27  priority  requirements,  the  products  also  addressed  many  of  the  other  17 
requirements.  Only  nine  requirements  did  not  have  any  applicable  technologies. 
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Table  2.  Prioritized  requirements  grouped  by  COI. 


Priority 

System  Requirement 

COI 

Technology 

1 

Develop  the  standard  desktop  client 
workstation  configuration  and  software  to 
allow  command/staff  organic  support  (CINC 

21 -configured  desktop  workstation). 

1  -  Decision- 
Focused 
Visualization 

EWS 

(not  assessed  directly) 

2 

Provide  domain-independent  mechanisms  for 
dynamically  tailoring  the  presentation  of 
information  to  a  decision-maker  based  upon 
their  activities. 

1  -  Decision- 
Focused 
Visualization 

DFC2  COA  Matrix 

FBV  AOR  Basing 
TCCC  Conseq.  Mgt. 

3 

Provide  domain-independent  mechanisms  for 
displaying  linkages  between  related 
events/items  within  and  across  windows, 
including  drill-down. 

1  -  Decision- 
Focused 
Visualization 

DFC2  COA  Matrix 

FBV  AOR  Basing 
Message  Tracker 

4 

Provide  generalized  XML-capable  browser 
and  integrate  with  the  Knowledge 

Management  (KM)  database. 

1  -  Decision- 
Focused 
Visualization 

DFC2 

STRATCOM  Portal 

5 

Provide  open,  extensible  geo-situation 
software  that  allows  acceptance  of  data  and 
geo-rendering  with  complete  filtering, 
labeling,  and  overlay  control. 

1  -  Decision- 
Focused 
Visualization 

XIS  provided  as  part 
of  AOR  Basing 

(not  assessed  directly) 

6 

Provide  the  standard  distributed  collaboration 
software  client  configuration  and  software  to 
allow  command/staff  organic  support  to 
users. 

3  -  Collaboration 

DCTS 

7 

Develop  a  thematic  collaboration  system  that 
ingests  information,  produces  summaries  and 
links  it  to  other  information  articles,  allows 
analysis  and  collaboration. 

3  -  Collaboration 

DFC2 

8 

Develop  a  solution  for  secure  authentication 
on  SIPRNET  among  the  COCOM’s  staff, 

Joint  Task  Forces  (JTFs),  and  Components. 

4  -  Security 

User  Management 

9 

Provide  tools  for  monitoring  and  visualizing 
information  flows  across  the  theater  based 
upon  content,  priority,  and  purpose. 

4  -  Enterprise 
Awareness 

DFC2  COA  Matrix 

FBV  AOR  Basing 

10 

Provide  tools  for  constructing  and  viewing  the 
Information  Assurance  COP. 

Not  addressed 

11 

Provide  unclassified  restricted  (e.g., 

NIPRNET)  conference  server  to  support 
multiple  distributed  collaboration  sessions 
internal  to  DoD. 

3  -  Collaboration 

DCTS 
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Table  2.  Prioritized  requirements  grouped  by  COI.  (continued) 


Priority 

System  Requirement 

COI 

Technology 

12 

Develop  a  secure  mechanism  that  enables 
collaboration  tools  to  work  in  the  presence  of 
firewalls. 

4  -  Security 

DCTS 

IWS 

VPN 

13 

Provide  a  secret  conference  server  to  support 
multiple  distributed  collaboration  sessions  with 
US  units. 

3  -  Collaboration 

DCTS 

IWS 

14 

Provide  unclassified,  unrestricted  conference 
server  to  support  multiple  distributed 
collaboration  sessions  with  organizations  and 
agencies  external  to  US  DoD. 

3  -  Collaboration 

DCTS 

15 

Field  collaboration  tools  to  the  COCOM,  JTF, 
and  Component  Commanders  that  provide 

VTC,  voice,  instant  messaging,  shared  virtual 
workspace,  shared  whiteboard,  and  shared 
applications. 

3  -  Collaboration 

DCTS 

IWS 

16 

Implement  a  solution  to  achieve  run-time 
interoperability  among  different  collaboration 
tools. 

3  -  Collaboration 

Collaboration  tools 

17 

Provide  mechanisms  for  automatically  setting 
up  and  managing  collaboration  sessions 
based  upon  workflow  and  operational 
priorities. 

3  -  Collaboration 

DFC2 

18 

Develop  a  solution  for  securely  connecting  the 
COCOM  to  his  Allied  counterparts  over 
unclassified  networks. 

4  -  Security 

VPN 

19 

Provide  secret  conference  server  to  support 
multiple  distributed  collaboration  sessions  with 
allies. 

3  -  Collaboration 

DCTS 

20 

Provide  mechanisms  for  automating  staging 
and  distribution  of  knowledge  objects 
throughout  the  theater  (integrate  with  IDM). 

2  -  Knowledge 
Management 

Enterprise  C2 
Infrastructure,  DFC2, 
COA  Matrix,  and 

COINS 

21 

Provide  the  semantic  constructs  required  for 
knowledge  exchange  for  critical  COCOM/JTF 
warfighting  and  theater  engagement 
processes. 

2  -  Knowledge 
Management 

DFC2 

22 

Provide  tools  that  enable  structured  KM 
publishing  processes  that  enable  producers  to 
publish  their  products  as  XML-based 
documents. 

2  -  Knowledge 
Management 

Enterprise  C2 
Infrastructure 
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Table  2.  Prioritized  requirements  grouped  by  COI.  (continued) 


Priority 

System  Requirement 

COI 

Technology 

23 

Build  a  knowledge  portal  for  viewing  and 
navigating  through  the  COCOM’s  Ops-lntel 

Brief,  SITREPs,  and  Electronic  Battlebook. 

1  -  Decision- 
Focused 
Visualization 

DFC2 

JWID  Admin  Portal 

24 

Provide  knowledge  portals  for  the  JTF  and 

Allies  that  enable  easy  access  to  COCOM- 
provided  support  services. 

1  -  Decision- 
Focused 
Visualization 

DFC2 

JWID  Portal 

25 

Implement  data  extraction  and  automated 
assembly  mechanisms  for  automating 
significant  portions  of  the  COCOM’s  morning 
brief. 

2  -  Knowledge 
Management 

DFC2  COA  Matrix 

RFI  Conseq.  Mgt. 
TCCC  Msg.  Tracker 

26 

Develop  automated  brief  preparation  capability 
based  upon  content  and  update  of  the 
information/knowledge  base. 

1  -  Decision- 
Focused 
Visualization 

DFC2 

Consequence  Mgmt. 

27 

Develop  consequence  management  and 
targeted  briefing  process. 

1  -  Decision- 
Focused 
Visualization 

DFC2 

Consequence  Mgmt. 

28 

Develop  the  standard  conference  room 
configuration  to  allow  command/staff  organic 
support  (CINC  21 -configured  conference 
room). 

3  -  Collaboration 

DCTS 

IWS 

29 

Develop  DTDs  for  OPLANs  and  CONPLANs. 

1  -  Decision- 
Focused 
Visualization 

Enterprise  C2 
Infrastructure; 

Knowledge  Core  and 
DFC2:  Decision  Space 
Data  Schema,  Schema 
Management 

30 

Develop  the  dynamic  TPFDD. 

Not  addressed 

31 

Implement  mechanisms  for  automating 
processes  consistent  with  the  workflow 
management  system  currently  under 
development  at  PACOM. 

1  -  Decision- 
Focused 
Visualization 

Task  Management 

System  (TMS) 

32 

Provide  next-generation  situation  awareness, 
planning,  and  force  execution  tools  to  the 
COCOM  and  integrate  with  KM  environment. 

4  -  Enterprise 
Awareness 

DFC2  COA  Matrix 

FBV  AOR  Basing 

33 

Provide  the  infrastructure  that  enables 
selected  portions  of  the  COCOM’s  networks  to 
support  prioritized  delivery  of  products  and  bit- 
streams. 

Not  addressed 
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Table  2.  Prioritized  requirements  grouped  by  COI.  (continued) 


Priority 

System  Requirement 

COI 

Technology 

34 

Field  advanced  protocols  to  significantly 
reduce  traffic  latency  of  selected  information 
flows  through  theater  networks. 

Not  addressed 

35 

Develop  a  creditable  solution  for  auto¬ 
generation  of  security  tags  for  archived  data 
products  based  upon  the  current  coalition 
release  policy. 

Not  addressed 

36 

Provide  high-resolution,  large-screen  displays 

1  -  Decision- 

PACOM  Video  Wall 

to  the  COCOM  that  are  capable  of  supporting 

Focused 

STRATCOM  K-Wall 

simultaneous  crises  and  day-to-day 
operations. 

Visualization 

37 

Develop  mechanisms  for  dynamically 

1  -  Decision- 

PACOM  Video  Wall 

managing  the  pixel  real  estate  of  large 

Focused 

STRATCOM  K-Wall 

displays  to  improve  comprehension. 

Visualization 

38 

Provide  alerting  mechanisms  that  auto-initiate 

4  -  Enterprise 

DFC2 

appropriate  visualization  events. 

Awareness 

FBV 

TCCC 

39 

Implement  homogeneous  server-server 

4  -  Enterprise 

Enterprise  C2 

federations. 

Awareness 

Infrastructure; 
Knowledge  Core  (XML 
Data  Services)  and 
DFC2  Schema 
management 

40 

Develop  the  standard  mobile  client  workstation 
configuration  and  software  to  allow 
command/staff  organic  support  (CINC  21- 
configured  mobile  workstation). 

Not  addressed 

41 

Provide  high-resolution,  large  displays  to  a 

4  -  Enterprise 

STRATCOM  K-Wall 

JTF  capable  of  supporting  simultaneous  crisis 
operations. 

Awareness 

PV290 

42 

Provide  a  method  for  accessing  knowledge 
objects  across  security  enclaves  with  minimal 
latency. 

Not  addressed 

43 

Prepare  for  the  transition  from  Ipv4  to  Ipv6  by 
performing  selected  experimentation  and  risk- 
reduction  activities  on  selected  theater 
networks. 

Not  addressed 

44 

Provide  multi-modal  human-computer 
interface  capabilities  (e.g.,  voice,  head¬ 
tracking,  gesture,  hand-pointing). 

Not  addressed 
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Who  Had  Assessment 
Responsibility? 


Organizational  Responsibilities 


The  principal  organizations  that  executed  CINC  21  were  Deputy  Under  Secretary  of  Defense 
(DUSD)  Advanced  Systems  &  Concepts  (AS&C),  United  States  Pacific  Command 
(USPACOM),  United  States  Strategic  Command  (USSTRATCOM),  Office  of  Naval  Research 
(ONR),  Defense  Information  Systems  Agency  (DISA),  SPAWAR  Systems  Command 
(SPAWAR),  DUSD  Science  &  Technology  (S&T),  Defense  Intelligence  Agency  (DIA),  and 
Defense  Threat  Reduction  Agency  (DTRA).  USPACOM  was  the  sponsoring  COCOM  and 
provides  the  Operational  Manager  for  the  ACTD.  ONR  and  DISA  shared  the  responsibilities  of 
Technical  Manager.  SPAWAR  acted  as  the  Transition  Manager  for  the  CINC  21  ACTD.  US 
PACCOM  and  USSTRATCOM  provided  the  operational  settings  for  the  ACTD. 

USSTRATCOM  provided: 

•  Participation  in  demonstrations/operations 

•  Input  on  military  utility  of  technologies 

USPACOM  provided: 

•  Operational  Manager 

•  On-site  technical  and  installation  support  to  Technical  Manager 

•  Constant  oversight  of  operational/user  requirements 

•  Concept  of  Operations  development 

•  Leadership  for  Joint  Military  Utility  Assessment  activities 

•  Participation  in  demonstrations  and  operations 


Operational  Manager 


The  user  sponsor,  USPACOM,  served  as  the  Operational  Manager.  The  Operational  Manager 
had  overall  responsibility  for  Concept  of  Operations  development,  requirements  definition,  and 
military  utility  assessment.  The  Operational  Manager  lead  the  planning,  execution,  and  reporting 
of  the  military  assessment  and  assisted  the  Technical  Manager  in  overall  ACTD  planning  and 
development  activities. 
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USPACOM  served  as  the  warfighter  sponsor  for  the  CINC  21  ACTD.  USPACOM  designated 
the  Chief  Information  Officer  (CIO)  as  the  Operational  Manager  for  CINC  21. 

Specific  responsibilities  included  the  following: 

•  Develop,  plan,  and  conduct  JMUA. 

•  Develop  metrics  to  measure  C2  effectiveness. 

•  Identify  opportunities  for  demonstrations. 

•  Report  JMUA  findings  and  recommend  for  transition  or  termination. 


Military  Utility  Assessment  Manager 


The  Military  Utility  Assessment  Manager  was  responsible  for  measuring  the  effectiveness  of  the 
ACTD  products  in  improving  operational  capabilities  and  reporting  the  findings  to  the 
Operational  Manager  along  with  a  recommendation  on  military  utility.  Military  utility  is  a 
judgment  of  the  military  worth  of  a  proposed  capability.  The  assessment  was  accomplished  by 
evaluating  performance  measured  in  an  operationally  realistic  environment  against  critical 
operational  issues.  The  assessment  considered  operational  effectiveness  and  suitability  in 
performing  the  assigned  mission  and  overall  importance  to  the  success  of  the  military  operations 
in  judging  military  worth.  The  Military  Utility  Assessment  Manager  was  responsible  to  the 
Operational  Manager  for  the  following: 

•  Identifying,  planning,  and  coordinating  assessment  opportunities. 

•  The  exercises,  demonstrations,  and  day-to-day  operations  provided  continuous 
opportunities  to  assess  CINC  2 1  technologies  for  military  value  and  provide  feedback  to 
the  technical  development,  integration,  and  transition  efforts. 

•  Developing  a  set  of  performance  measures,  an  approach  to  provide  maximum  feedback, 
ensured  technologies  were  assessed  quickly. 
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How  Was  CINC  21  Assessed? 


General  Approach 


Based  on  the  Ultimate  Operational  Issue  (UOI),  COIs,  CINC  21  objectives,  and  prioritized 
requirements,  a  JMUA  plan  (Appendix  B  )  was  developed  to  evaluate  CINC  2 1  capabilities  for 
acquiring,  processing,  analyzing,  and  presenting  information  needed  for  better  decision-making, 
improved  processes,  increased  overall  readiness,  efficiency,  and  mission  effectiveness.  Measures 
of  effectiveness  (MOEs)  were  defined  along  with  subsequent  measures  of  perfonnance  (MOPs) 
and  metrics  to  assess  the  CINC  2 1  products  in  an  operational  setting. 


MOE  Development  Process 


After  several  iterations,  an  initial  JMUA  plan  was  developed  in  January  2001  (Appendix  B).  This 
plan  summarized  an  approach  and  proposed  a  group  of  metrics  for  examining  how  well  the 
CINC  21  technologies  support  the  Combatant  Commander  information  and  decision  processes. 
After  development  and  integration  of  many  of  the  CINC  2 1  technologies,  mission  areas  were 
defined  as  the  context  for  compiling  a  more  detailed  JMUA  plan. 

A  CINC  21  Assessment  Workshop  was  held  in  San  Diego,  CA,  on  23-25  January  2002.  While 
hosted  and  chaired  by  the  JMUA  team,  attendees  included  all  the  major  participants  of  the  CINC 
2 1  ACTD:  the  Deputy  Technical  Manager,  the  Deputy  Operational  Manager,  the  Military  Utility 
Assessment  Manager,  the  Transition  Manager,  the  development  trust  leaders,  and  representatives 
from  the  technical  development  team.  The  participants  included  representation  from  USPACOM 
and  USSTRATCOM.  Work  groups  proposed  a  number  of  information  requirements  as 
components  of  the  JMUA,  based  on  the  planned  technology  products  of  CINC  21.  The  major 
outcome  of  the  workshop  was  a  set  of  MOEs  that  expanded  the  COIs. 

For  each  COI,  one  or  more  MOEs  that  transform  the  COI  into  a  statement  of  measurability  were 
defined.  The  MOEs  are  high-level  statements  that  summarize  the  extent  to  which  a  system 
accomplishes  or  supports  a  mission  or  task,  and  are  designed  to  address  an  individual  COI. 

MOEs  are  not  necessarily  exhaustive.  Rather,  a  comprehensive  set  of  MOEs  that  adequately 
address  the  expected  improvements  by  each  group  of  deliverables  has  to  be  defined.  Each  MOE 
is  defined  in  more  detail  by  one  or  more  MOPs.  MOPs  are  qualitative  or  quantitative  measures  of 
a  system’s  capabilities  or  specific  performance  function.  Finally,  metrics  are  defined  as  specific 
measures  used  to  address  the  MOPs. 


25 


Joint  Military  Utility  Assessment 

The  process  used  in  defining  assessment  requirements  and  then  transforming  these  into  specific 
measurements,  is  shown  in  Figure  3.  The  assessment  targets  are  the  final  deliverables,  as  defined 
by  the  CINC  21  ACTD. 
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Figure  3.  Defining  assessment  requirements  and  developing  specific  measures. 


Military  Effectiveness  Criteria 


Comprehensive  assessment  involves  layers  of  measurement,  including  the  determination  that  (1) 
enabling  hardware  and  software  function  adequately;  (2)  personnel  were  adequately  trained  to 
operate  new  technology,  and  can  use  the  technology  to  do  their  work;  and  (3)  performance  of 
key  selected  functions  or  part-tasks  has  improved  as  expected.  Consequently,  as  shown  in  Figure 
4,  the  Joint  Military  Utility  Assessment  ( JMUA)  addresses  the  following  effectiveness  criteria: 

•  Suitability  -  Are  operational  requirements  met? 

•  Usability  -  Are  CINC  2 1  products  easy  to  learn  and  use? 

•  Technical  foundation 

o  Supportability  -  Are  the  products  accommodated  within  the  existing  infrastructure? 
o  Accreditability  -  Are  services  of  CINC  2 1  products  exportable? 
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Figure  4.  Military  effectiveness  criteria  for  assessing  hardware  and  software. 


Joint  Military  Utility  Assessment  Plan 


Following  the  Assessment  Workshop,  the  assessment  team  compiled  a  comprehensive  set  of 
MOEs  that  integrated  the  measurement  requirements  of  the  various  work  groups.  Thirteen  MOEs 
were  defined  in  terms  of  quality,  time,  efficiency,  and  network  operations: 

Quality 

•  Quality  of  situation  awareness  and  understanding  among  decision-makers  is 
improved. 

•  Quality  of  decision-making  is  improved. 

•  Quality  of  collaborative  operations  is  improved. 

•  Quality  of  collaborative  planning  is  improved. 

•  Quality  of  infonnation  is  improved. 

Time 

•  Time  it  takes  to  reach  decisions  is  decreased. 

•  Time  to  collaboratively  plan  is  decreased. 

•  Time  to  relay  information  to  decision-makers  is  decreased. 

Efficiency 

•  Efficiency  of  information-sharing  among  distributed  operations  is  improved. 

•  Efficiency  of  work  processes  is  improved. 

•  Efficiency  transmission  of  information  among  distributed  operations  is  improved. 
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Network  Operations 

•  C2  infrastructure  provides  security  assurance  across  multiple  networks,  databases, 
and  applications. 

•  Servers  and  applications  are  compatible  and  interoperable  with  existing  services, 
applications,  and  network  configurations. 

The  MOEs  were  mapped  to  the  COIs  to  ensure  sufficient  coverage.  Although  some  MOEs 
applied  to  more  than  one  COI,  Table  3  provides  a  summary  of  how  the  MOEs  aligned  with  the 
COIs.  These  MOEs,  with  subordinate  MOPs  and  metrics,  were  published  in  the  CINC  21 
Military  Utility  Assessment  Plan  for  FY  02/03  (Appendix  B).  The  original  MOEs  have 
functioned  with  few  changes  as  the  foundation  of  data  collection  efforts  for  the  project,  despite 
changes  in  the  direction  of  development  efforts.  A  spreadsheet  of  these  measurement  criteria  and 
data  collected  is  located  in  Appendix  C. 


Assessment  Methods  and  Events 


CINC  2 1  assessment  efforts  were  conducted  throughout  the  technology  development  and 
integration  period.  Consequently,  different  sets  of  assessment  methods  and  metrics  from  the 
JMUA  Plan  were  employed  as  appropriate  for  the  technology  type,  development  maturity,  and 
assessment  environment.  Trouble  reports  and  technical  observations/logs  were  analyzed  to 
identify  successes  and  problems  associated  with  the  installation  and  integration  of  the  diverse  set 
of  CINC  21  products.  The  usability  of  CINC  21  products  was  assessed  via  several  methods:  (1) 
heuristic  evaluations  that  compared  the  user  interface  against  industry  standards  and  guidelines, 
(2)  informal  comments  and  observations  by  representative  users,  and  (3)  structured  ratings  and 
performance  tests  of  the  user  interface  under  scenario-based  task  conditions.  The  impact  of 
CINC  2 1  products  on  organizational  effectiveness  and  work  processes  was  assessed  in  similar 
ways.  The  suitability  of  CINC  2 1  technology  to  support  operational  tasks  was  assessed  via  direct 
involvement  by  representative  operational  decision-makers  and  staff.  They  provided  comments 
and  ratings  on  the  technology  during  several  demonstrations  and  training  sessions,  and  they  were 
able  to  use  the  technology  for  their  dynamic  tasks  in  large-scale  exercises. 

In  general,  however,  three  types  of  assessment  venues  were  used  throughout  the  JMUA: 

•  Military  Exercises  and  Demonstrations 

•  Limited  Assessment  Opportunities 

•  Technical  Evaluations  (includes  human-computer  interface  evaluations) 

Assessment  data  were  collected  at  three  types  of  events:  military  exercises/demonstrations, 
limited  assessment  opportunities  (LAOs),  and  human-computer  interface  (HCI)  evaluations  by 
expert  human  factors  reviewers.  Table  4  shows  the  specific  CINC  21  technologies  that  were 
assessed  during  various  military  exercises  or  demonstrations. 
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Table  3.  COI  to  MOE  mapping 


COI 

MOE 

COM:  Can  advanced  visualization  technology 
empower  individuals  to  process,  digest,  and 
assimilate  large  volumes  of  information,  enabling 
faster,  more  effective  decisions 

Quality 

Quality  of  situation  awareness  and  understanding  among 
decision-makers  is  improved. 

Quality  of  decision-making  is  improved. 

COI-2:  Can  knowledge  management  technology 
integrate  information,  context,  and  rules  to 
increase  understanding  and  improve  decision¬ 
making? 

Time 

Time  it  takes  to  reach  decisions  is  decreased. 

COI-3:  Can  collaboration  tools  be  used  to 
overcome  tyrannies  of  time,  distance,  and 
system  disparity? 

Quality 

Quality  of  situation  awareness  and  understanding  among 
decision-makers  is  improved. 

Quality  of  collaborative  operations  is  improved. 

Quality  of  collaborative  planning  is  improved. 

Time 

Time  it  takes  to  reach  decisions  is  decreased. 

Time  to  collaboratively  plan  is  decreased. 

Efficiency 

Efficiency  of  information  sharing  among  distributed 
operations  is  improved. 

COM:  Can  the  collection  of  networks, 
databases,  and  applications  be  enhanced  to 
optimize  the  flow  of  information,  with  security 
assurance,  across  multiple  network  enclaves? 

Quality 

Quality  of  information  is  improved. 

Time 

Time  to  relay  information  to  decision-makers  is 
decreased. 

Efficiency 

Efficiency  of  work  processes  is  improved. 

Efficiency  transmission  of  information  among  distributed 
operations  is  improved. 

Network  Operations 

C2  infrastructure  provides  security  assurance  across 
multiple  networks,  databases,  and  applications. 

Servers  and  applications  are  compatible  and 
interoperable  with  services,  applications,  and  network 
configurations. 
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Large-scale  military  exercises  and  demonstrations  afforded  opportunities  for  the  CINC  21 
technologies  to  be  used  in  high-fidelity  tasks.  Users  and  operational  personnel  who  view  the 
technologies  “in  action”  can  make  pragmatic  judgments  of  their  military  utility.  In  addition, 
usability  and  technical  issues  can  be  assessed. 

LAOs  are  venues  that  are  focused  on  a  specific  technology  or  capability.  They  are  much  more 
limited  in  scope  than  exercises  or  demonstrations,  and  they  allow  much  tighter  control  of  the 
assessment  process.  The  task  context  is  situationally  specific.  LAOs  afford  assessment  personnel 
the  opportunity  to  observe  in-depth  interactions  between  operational  users  and  the  technology 
and  to  gather  detailed  feedback  regarding  utility  and  usability  issues.  Assessment  data  were 
collected  at  LAOs  for  the  following  technologies:  Decision-Focused  Command  and  Control — 
Decision  Space  Management,  Decision  Summary,  Chat,  Battle  Rhythm,  Status  Rule 
Management,  and  NetOps;  AORB;  Consequence  Management  Automated  Brief;  RFI  Manager; 
Message  Tracker;  USSTRATCOM  Knowledge  Wall;  Collaboration;  TMS;  and  Master  Calendar. 


Table  4.  CINC  21  Technologies  at  military  exercises  and  demonstrations. 


Military  Exercise  or  Demonstration 

CINC  21  Technology  Assessed 

Build  1  Technologies  Demonstration  Oct  00 

Collaboration,  Infrastructure  and  Security 

Kernel  Blitz  Experimental  KB(X)  Jun  01 

TeamApp,  Collaboration 

Joint  Warrior  Interoperability  Demonstration 
(JWID)  May  02 

JWID  Administrative  Portal,  JWID  Coalition 
Interoperability  Service  (COINS), 

Collaboration,  Infrastructure  and  Security 

USSTRATCOM  Consequence  Management/ 
Response  Demonstration  Oct  02 

FBV,  USSTRATCOM  portal,  USSTRATCOM 
Knowledge  Wall,  Collaboration,  COA  Matrix 

Terminal  Fury  '03  (TF03)  Dec  02 

Decision-Focused  Command  and  Control: 
Decision  Space  Management,  Decision 

Summary,  Chat,  Battle  Rhythm,  Status  Rule 
Management 

Technology  experts  familiar  with  the  design  and  configuration  of  the  technologies  conducted 
technical  evaluations.  Human  factors  experts  assessed  human-computer  interfaces  using  the 
CINC  2 1  Style  Guide  or  Heuristic  Evaluation-A  System  Checklist.  The  following  technologies 
were  assessed:  Decision-Focused  Command  and  Control — Decision  Space  Management  and 
Status  Rule  Management;  Consequence  Management  Automated  Brief;  RFI  Manager;  Message 
Tracker;  USSTRATCOM  Knowledge  Wall;  Multi-panel  Display;  JWID  Administrative  Portal; 
TMS;  and  Master  Calendar.  Computer  systems  experts  evaluated  the  functioning  of 
infrastructure  and  security  systems. 


1  Pacific  Science  &  Engineering  Group.  2002.  User  Interface  Design  Guidelines  for  Web  Portals.  San  Diego,  CA. 
3  D.  Pierotti.  2001.  Heuristic  Evaluation-A  System  Checklist.  Xerox,  Palo  Alto,  CA. 
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Results  from  individual  assessment  events  were  reported  as  they  occurred  throughout  the  CINC 
21  project  period.  Many  of  these  reports  have  been  compiled  and  attached  as  appendices  to  this 
report.  See  the  appendices  to  read  the  assessment  of  a  specific  CINC  2 1  product  within  a 
particular  assessment  environment. 


Instruments 


A  wide  variety  of  instruments  were  used  to  collect  data  during  exercises,  demonstrations,  and 
limited-assessment  opportunities,  as  shown  in  Table  5. 

Selections  of  these  instruments  were  adapted  for  the  purpose  of  each  individual  assessment 
event,  as  shown  in  the  assessment  documents  presented  in  the  appendices. 
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Table  5.  Descriptions  of  data  collection  instruments 


Instrument 

Description 

Training  Assessment 

Form 

A  form  was  developed  for  administration  at  the  completion  of  training 
to  determine:  (1 )  if  sufficient  training  was  received  on  the  features  of 
each  technology,  (2)  the  tasks  to  which  the  technology  would  be 
applied,  and  (3)  if  sufficient  training  had  been  received  to  apply  the 
technology  to  work  tasks. 

Operational  Utility 

Each  of  the  CINC  21  technologies  were  demonstrated  or  used.  The 
users  responded  to  questionnaires  to  assess  the  degree  of 
operational  utility. 

Attitude  Rating 

The  attitude  measurement  uses  a  technique  of  semantic  differential, 
bi-polar  adjective  scales.  The  attitude  form  used  four  scales,  with  one 
selected  from  each  group.  These  were  (1)  a  Useless-Useful  scale,  (2) 
a  Powerless-Powerful  scale,  (3)  a  Slow-Fast  scale,  and  (4)  a 
Chaotic-Orderly  scale. 

Human-Computer 

Interface  Rating 

A  series  of  eight  scales  were  selected  that  highlight  major  features  of 
the  interface  between  a  human  and  the  computer  (Human-Computer 
Interface  (HCI)).  The  eight-scale  HCI  instrument  was  intended  to  be 
administered  twice  to  each  participant,  as  critical  design  details  may 
not  be  noticed  until  extensive  use. 

Situation  Assessment 

Rating  Technique  (SART) 

Situation  awareness  (SA)  is  the  experience  of  fully  understanding 
what  is  going  on  in  a  given  situation,  of  seeing  each  element  within  the 
context  of  the  overall  mission  or  goal,  and  of  having  all  the  pieces  fit 
together  into  a  coherent  picture.  The  Situation  Awareness  Rating 
Technique  (SART)  is  a  subjective  SA  method  developed  by  Selcon  & 

T aylor  (1 989)4  for  the  UK  Defense  Evaluation  and  Research  Agency. 

Feature  Utility 

After  using  a  technology,  the  participants  could  provide  an 
assessment  of  the  utilization  of  each  of  the  technology’s  features  for 
accomplishing  their  work.  A  one-page  form  was  provided  for  each 
technology  used  by  a  participant,  with  fields  to  make  a  judgment  of 
each  feature  with  regard  to  (1)  Needs  to  be  fixed,  (2)  Mixed  reaction, 

(3)  Great  as  is,  and  (4)  Not  Used. 

Task  Workload  Index 
(TLX) 

The  NASA  Task  Load  Index  is  a  thoroughly  studied  (Byers,  Bittner, 

Hill,  Zaklad,  &  Christ,  1988)5  multi-dimensional  subjective  rating 
procedure  that  affords  an  overall  workload  score  based  on  the 
weighted  average  of  these  six  subscale  ratings:  Mental  Demands, 
Physical  Demands,  Temporal  Demands,  Performance,  Effort,  and 
Frustration.  In  addition  to  the  six  scale  scores,  more  typically  an 
overall  weighted  measure  of  task  load  can  be  calculated  based  on  the 
subscales. 

4  S.  J.  Selcon  and  R.  M.  Taylor.  1989.  “Evaluation  of  the  Situational  Awareness  Technique  (SART)  as  a  Tool  for 
Aircrew  Systems  Design.”  In  Proceeding  of  the  A  GARD  AMP  Symposium  on  Situational  Awareness  in  Aerospace 
Operations.  Advisory  Group  for  Aerospace  Research  and  Development,  Neuilly  Sur  Seine,  France. 

5  J.  C.  Byers.  A.  C.  Bittner,  Jr.,  S.  G.  Hill,  A.  L.  Zaklad,  and  R.  E.  Christ.  1988.  “Workload  Assessment  of  a 
Remotely  Piloted  Vehicle  (RPV)  System.”  Proceedings  of  the  Human  Factor  Society  32nd  Annual  Meeting 
(pp.  1 145-1 149).  Human  Factors  and  Ergonomics  Society,  Santa  Monica,  CA. 
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Table  5,  continued. 

Instrument 

Description 

Military  Utility  Assessment 
Questionnaire 

Measures  of  Performance  associated  with  each  Measure  of 

Effectiveness  was  stated  as  a  question,  and  included  in  a 
questionnaire  to  be  answered  the  participants. 

Baseline  Measures 

Estimates  of  frequency,  quantity,  quality  and  timing  were  made  by 
subject  matter  experts  for  current  operations,  to  be  used  for 
comparison  with  new  and  alternative  methods. 

Australian  Assessment 

Tool  (A AT) 

A  web-based  tool  was  used  to  collect  data  from  participants  in 
response  to  questions  about  their  experience  and  specific  questions 
generated  by  the  study  team.  These  data  were  entered  through 
computer  workstations  at  US-USPACOM,  Canada,  Australia,  and  UK. 
The  data  were  collected  into  an  integrated  database  for  subsequent 
analysis. 

Joint  Battle  Center  Data 
Collection  and  Analysis 

Tool  (JD-CAT) 

A  web-based  tool  was  used  to  collect  data  from  participants  in 
response  to  questions  about  their  experience  and  specific  questions 
generated  by  the  study  team.  These  data  were  entered  through 
computer  workstations  at  US-USPACOM,  Canada,  Australia,  and  UK. 
The  data  were  collected  into  an  integrated  database  for  subsequent 
analysis. 

Observer  (third  party)  data 
collection  forms 

Data  collection  forms  were  designed  for  observer  data  collectors  to 
collect  timing  of  specific  events,  observations  associated  with 
anticipated  and  unanticipated  events,  and  impromptu  participant 
verbal  responses. 

Interview  forms 

Measures  of  Performance  associated  with  each  Measure  of 

Effectiveness  was  stated  as  a  question  and  used  as  the  basis  for 
interviews  with  participants  and  individuals  who  watched 
demonstrations  of  the  technology 

Style  Guide  Checklist 

A  CINC  21  Style  Guide  was  developed  from  a  wide  range  of 
applicable  human  factors  guidelines.  A  checklist  was  developed  so 
that  a  human  factors  expert  could  rate  an  interface  for  each  section  of 
the  style  guide,  using  a  scale  of  1  to  7. 

Heuristic  Checklist 

A  Heuristic  Checklist  was  compiled  based  on  14  rules  (heuristics)  that 
have  been  identified  as  characteristic  of  usable  systems  (Nielsen, 

1993)6.  Human  Factors  experts  and/or  users  could  rate  a  component 
of  CINC  21  for  each  heuristic  on  a  scale  of  1  -to  7. 

J.  Nielsen.  1993.  Usability  Engineering.  Academic  Press,  Boston,  MA. 
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Participants 


Active  duty  military  personnel  and  civilian  personnel  served  as  participants  for  CINC  21 
technology  assessment  at  the  military  exercises,  demonstrations,  and  LAOs.  Personnel  affiliated 
with  USSTRATCOM,  USPACOM,  First  Marine  Expeditionary  Force  (I  MEF),  and  Commander 
Third  Fleet  comprised  the  participant  pool.  In  addition,  for  the  JWID,  active  duty  military 
participants  from  Australia,  Canada,  and  the  United  Kingdom  were  involved. 

For  the  LAOs,  participants  were  active  duty  military  personnel,  civilian  personnel  with 
background  as  military  operational  personnel,  civilian  personnel  working  in  staff  positions  at 
USPACOM  or  USSTRATCOM,  and  civilian  personnel  working  as  researchers  at  Space  and 
Naval  Warfare  Command  Center,  San  Diego.  CINC  21  assessment  personnel  were  on  hand  to 
train  and  assist  users  to  complete  the  appraisal  tasks. 


Statistical  Analyses 


When  collecting  data  with  human  subjects,  remember  that  results  vary  from  person  to  person, 
and  from  time  to  time.  Thus,  the  mean  of  data  collected  for  Device  A  can  differ  from  the  data  for 
Device  B  by  chance,  and  consequently  it  is  desirable  to  know  the  probability  that  such 
differences  could  occur  by  chance.  However,  one  should  know  that  it  takes  a  quantity  of  data 
(repeated  measurements)  to  derive  a  good  estimate  of  this  probability.  Where  possible  with  the 
data  from  various  assessment  events,  statistical  tests  were  perfonned  as  shown  in  Table  6.  The 
reader  is  referred  to  the  detailed  appendices  for  more  information  about  how  these  tests  were 
used  to  assess  specific  technologies. 

User  responses  to  questionnaire  items  provided  the  bulk  of  the  structured  assessment  data. 
Assessment  data  were  analyzed  mainly  through  descriptive  statistics.  In  most  of  the  assessment 
events,  the  number  of  participants  was  not  large  enough  to  warrant  the  use  of  inferential 
statistics.  The  exceptions  were  data  from  KB(X),  where  analysis  of  variance  was  used  to  analyze 
group  differences  in  response  to  Situation  Assessment  Rating  Technique  (SART)  scale  items, 
and  data  from  JWID,  where  chi-square  analyses  were  applied  to  determine  whether  the  results 
(responses  to  online  Joint  Battle  Center  Data  Collection  and  Analysis  Tool  [JD-CAT]  and  the 
Australian  Assessment  Tool  [AAT]  questionnaire  items)  could  have  occurred  randomly  more 
often  than  one  time  out  of  twenty.  Content  analysis  was  used  to  distill  interview,  user  comment, 
and  observational  data. 
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Table  6.  Descriptions  of  statistical  tests  used  during  JMUA. 


Statistical  Test 

Description 

t-test  for  related  measures 

Test  of  the  significance  of  the  difference  between  two  means, 
for  example,  ratings  by  the  same  individuals  for  two  or  more 

CINC  21  technologists.  The  statistical  test  yields  the  probability 
that  the  difference  between  the  two  means  could  be  a  chance 

occurrence. 

Chi-Square  test 

For  frequency  data  (e.g.,  counts  of  responses),  the  observed 
frequency  can  be  compared  to  an  expected  frequency  (e.g., 
frequency  expected  for  no  effect/change/improvement).  The 
statistical  test  yields  the  probability  that  the  effect/change 
/improvement  was  real  rather  than  chance. 

Analysis  of  Variance  (ANOVA) 

For  selected  data  (e.g.,  Situation  Awareness  Rating  Test),  a 
statistical  test  could  be  conducted  that  compared  the  effects  of 
multiple  factors  in  the  same  analysis.  Thus,  the  significance  of 
each  factor,  and  the  interactions  among  factors,  could  be  tested 
for  chance  occurrence. 

Correlation  Analysis 

For  selected  data  (e.g.,  Workload)  a  test  was  performed  to 
determine  if  the  results  varied  early  versus  late  in  an  exercise. 

35 


Cll 


Joint  Military  Utility  Assessment 


Outcome 


Data  Collection  Summary 


The  JMUA  Plan  identified  an  array  of  MOEs,  MOPs,  and  metrics  organized  by  mission 
effectiveness  criteria  that  address  the  COI.  Table  7  summarizes  how  many  of  these  measures 
were  collected  at  some  point  in  the  multi-method  assessment  process7.  It  refers  only  to  the 
quantity,  not  the  quality  of  data  collected.  This  table,  then,  reports  the  breadth  of  the  overall 
JMUA  effort.  Details  are  provided  in  the  CINC  21  Products  Technology  Categories  Section. 

Table  7.  Number  of  MOEs,  MOPs,  and  metrics  collected  for 
Suitability,  Usability,  and  Technical  groupings. 


MEASURE 

MISSION  EFFECTIVENESS  CRITERIA 

TOTAL 

Suitability 

Usability 

Technical 

Supportability 

Accreditability 

“Are  operational 

requirements 

met?” 

“Are  CINC  21 
products  easy  to 
learn  &  use?” 

“Are  CINC  21  products 
accommodated  within  the 
existing  infrastructure?” 

“Are  services  of  CINC 

21  products 
exportable?” 

MOEs* 

9  of  9 

6  Of  6 

4  of  4 

1  Of  1 

20  of  20 

MOPs 

26  of  33 

12  of  12 

6  Of  6 

3  of  4 

47  of  55 

Metrics 

42  of  58 

25  of  25 

10  of  10 

5  of  11 

82  of  104 

*  There  are  13  MOEs — some  apply  to  more  than  one  category.  MOPs  and  metrics  are  specific  to  a  category. 


There  are  13  MOEs.  Data  relevant  to  each  of  the  13  MOEs  were  obtained.  The  MOEs  define 
desired  improvements  in  output  quality,  time  to  produce  output,  efficiency  of  processes,  or 
improvements  to  the  C2  infrastructure.  Subordinate  to  the  MOEs,  there  are  104  metrics 
contained  in  55  MOPs.  Definitions  of  the  MOEs  and  a  summary  of  the  MOPs  (47  of  55)  and 
metrics  (82  of  104)  for  which  data  were  obtained  are  shown  in  Table  8. 

A  coding  scheme  was  assigned  to  the  measurement  criteria  to  facilitate  their  use  during  the 
design  and  execution  of  data  collection  efforts.  That  is,  it  was  easier  for  assessors  to  refer  to 
“metric  Al-1”  rather  than  “measurement  of  accuracy  in  understanding  the  operational  situation” 
or  “the  first  metric  under  the  first  MOP  under  the  MOE  that  measure  quality  of  situation 
awareness  and  understanding  among  decision-makers.”  The  letters  and  numbers,  therefore,  have 
purpose  or  meaning  only  for  identification  of  the  measurement  criteria.  MOEs  were  assigned 
letters  (e.g.,  MOE  A).  MOPs  were  designated  by  the  letter  of  the  associated  MOE  plus  a  number 


7  For  details,  see  Appendix  C. 
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(e.g.,  MOP  Al,  MOP  A2).  Metrics  followed  the  same  pattern,  being  assigned  numbers 
consecutively  as  metrics  were  developed  (e.g.,  metric  Al-1,  metric  Al-2). 

Table  8.  MOE  definitions. 


MOE 

Definition 

MOPs 

measured 

Metrics 

measured 

A 

The  quality  of  situation  awareness  and  understanding  among 
decision-makers  is  improved. 

4  of  4 

9  of  10 

B 

The  time  to  reach  decisions  is  decreased. 

3  of  4 

4  of  5 

C 

The  quality  of  decisions  is  improved. 

2  of  3 

4  of  7 

D 

The  efficiency  of  information  sharing  among  distributed  operations  is 
improved. 

2  of  2 

2  of  2 

E 

The  quality  of  collaborative  operations  is  improved. 

4  of  7 

9  of  17 

F 

The  quality  of  collaborative  planning  is  improved. 

3  of  5 

6  of  8 

G 

The  time  to  collaboratively  plan  is  decreased. 

2  of  2 

2  of  3 

H 

The  quality  of  information  is  improved. 

7  of  7 

9  of  9 

1 

The  efficiency  of  work  processes  is  improved. 

8  of  8 

1 1  of  1 1 

J 

Efficiency  of  transmissions  of  information  among  distributed 
operations  is  improved. 

4  of  4 

1 1  of  1 1 

K 

C2  infrastructure  provides  security  assurances  across  multiple 
networks,  databases,  and  applications. 

4  of  5 

9  of  15 

L 

The  time  to  relay  information  to  decision-makers  is  decreased. 

2  of  2 

4  of  4 

M 

Servers  and  applications  are  compatible  and  interoperable  with 
existing  services,  applications,  and  network  configurations. 

2  of  2 

2  of  2 

MOPs  for  which  no  data  were  collected  are  listed  in  Table  9. 

With  one  exception,  the  MOPs  for  which  no  data  were  collected  are  subordinate  to  MOEs  in  the 
area  of  quality  of  output.  Basically,  the  design  of  the  assessment  venues  precluded  the  ability  to 
gather  data  that  would  apply  to  these  measures,  as  noted  below: 

•  Tasks  were  very  scripted  (impacted  C3,  E5,  E6,  E7,  FI,  F5). 

•  Tasks  relevant  to  the  MOP  were  not  performed  (impacted  B4). 

•  Specific  metrics  for  MOP  K4  were  never  defined. 
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Table  9.  MOPs  with  no  associated  assessment  data. 


MOP 

Definition 

C3 

The  ability  to  recognize  the  need  to  make  decisions  improved. 

E5 

Quality  of  product  was  improved  by  collaboration. 

E6 

Team  actions  were  better  synchronized  by  collaboration. 

E7 

The  team  was  more  agile  because  of  collaboration. 

FI 

Collaboration  improved  the  quality  of  the  plan  produced. 

F5 

The  quality  of  the  COA  alternative  set  improved. 

B4 

The  time  to  stand  up  the  JTF  was  faster. 

K4 

Command  of  the  information  enterprise  was  improved. 

There  are  22  metrics  that  were  not  used.  Eight  of  these  are  subsumed  under  the  MOPs  described 
above.  The  remaining  metrics  with  no  data  are  listed  in  Table  10. 

The  inability  to  measure  these  metrics  was  due  to  the  following: 

•  Tasks  were  very  scripted  (impacted  Al-1,  C2-1,  C2-2,  E2-2,  E2-3,  E2-4,  Gl-1).  In 
addition,  these  types  of  metrics  are  more  appropriate  for  comparison  of  intact,  fully 
functioning  teams  with  a  history  of  perfonning  these  tasks  so  that  comparisons  can  be 
made  of  baseline  performance  with  performance  using  the  new  technology. 

•  Tasks  relevant  to  the  metric  were  not  perfonned  (impacted  E4-5,  E4-6,  Kl-1,  Kl-2). 

Measurement  of  some  metrics  was  beyond  the  scope  of  the  assessment  (impacted  Kl-4,  Kl-6, 
K3-2). 
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Table  10.  Metrics  with  no  associated  assessment  data. 


Metric 

Definition 

C2-1 

Rating  of  the  extent  to  which  the  decision  process  conformed  with  consideration 
of  critical  uncertainties  (expert  judgment,  answer  key) 

C2-2 

Rating  of  the  decision  for  completeness  and  appropriateness  against 
organizational  criteria. 

E2-2 

Common  understanding  of  team  goals. 

E2-3 

Commonality  and  accuracy  in  understanding  independent  players  and  team 
tasks. 

E2-4 

Number  of  times  that  key  players  take  an  action  that  undermines  the  tasks  of 
another  player.  Number  (fraction)  of  times  that  key  players  take  an  action  that 
supports  the  task  of  other  key  players. 

E4-5 

Time  to  recognize  need  for  a  change  in  plan. 

E4-6 

Time  to  recognize  need  for  a  decision. 

G1-1 

Start  and  stop  times  required  to  develop  mission  and  intent. 

K1-1 

Number  of  successful  controller  attempts  to  monitor  network  operations  vs.  total 
controller  attempts  to  monitor  network  operations. 

K1-2 

Number  of  detected  intrusion  attempts  vs.  total  intrusion  attempts. 

K1-4 

Ability  of  controllers  to  identify  redundant  data  transmission  paths. 

K1-6 

System  administration  requirements  stay  within  acceptable  workload 
parameters. 

K3-2 

Number  of  applications/service  failures  vs.  total  attempts  to  access 
applications/services. 

Assessment  Summaries 


This  report  attempts  to  aggregate  across  each  of  these  individual  assessments  to  provide  an 
integrated  summary  of  military  utility.  We  realize,  however,  that  readers  will  need  different  types 
of  assessment  summaries,  aggregated  in  different  ways.  One  way  in  which  we  report  the 
assessment  data  is  in  terms  of  the  CINC  2 1  products  themselves.  For  ease  of  presentation,  we 
have  summarized  these  by  technology  categories.  These  assessment  summaries  reflect  the 
measures  of  effectiveness,  measures  of  performance,  and  associated  metrics  identified  in  the 
CINC  2 1  JMUA  Plan.  In  addition,  there  is  a  brief  discussion  of  technical  issues,  tradeoffs,  and 
lessons-leamed  during  the  installation  and  integration  of  the  products  and  infrastructure.  The 
assessment  data  have  also  been  summarized  in  terms  of  the  prioritized  CINC  21  system 
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requirements  and  capabilities.  Each  of  these  views  of  the  assessment  data  has  been  combined  to 
show  their  relationship  to  the  four  CINC  21  COIs. 

The  military  utility  assessment  of  CINC  2 1  technology,  then,  is  seen  as  a  multi-dimensional 
hierarchy.  The  most  elemental  assessments  of  CINC  2 1  products  in  particular  tests  or  exercises 
can  be  combined  to  provide  summary  evaluations  of  various  technology  categories,  system 
requirements,  and  other  dimensions.  Ultimately,  these  evaluations  may  be  aggregated  to  provide 
a  summary  assessment  for  each  COI.  Of  course,  some  degree  of  subjectivity  is  necessary  to 
make  these  aggregate  judgments.  This  subjectivity  has  been  controlled  by  cross-referencing  back 
to  the  fundamental  test  data  and  by  seeking  input  from  all  CINC  21  stakeholders  in  reporting 
these  summary  assessments. 

Throughout  this  document,  non-textual  summary  indicators  using  the  stoplights  metaphor  are 
provided  at  four  levels: 

1.  The  individual  technologies. 

2.  Technology  categories — groupings  of  individual  technologies. 

3.  COIs — groupings  of  technology  categories. 

4.  Prioritized  Requirements — Operations  Manager-identified  stipulations  for  advanced 
information  technology 

At  each  of  these  levels,  a  rating  is  provided  for  the  following  issues: 

■  Suitability — Are  operational  requirements  met? 

■  Usability — Is  the  technology  easy  to  learn  and  use? 

■  Technical — Are  CINC  21  technologies  accommodated  within  the  existing  infrastructure, 
and  can  their  services  be  securely  exported? 

These  summary  indicators  are  meant  to  provide  the  reader  with  a  means  to  scan  the  results. 

Much  detail  is  necessarily  lost  in  these  summaries  as  myriad  data  points  are  combined  into  a 
three  point  scale;  it  is,  therefore,  strongly  suggested  that  readers  do  not  rely  on  these  summary 
ratings  without  a  thorough  review  of  the  finer  level  analyses. 

The  ratings  at  the  summary  indicators  were  attained  by  applying  the  following  guidelines: 

Green:  A  preponderance  of  the  MOPs  were  rated  as  Positive  Findings  rather  than  as  Remaining 
Challenges,  AND  any  Remaining  Challenges  are  judged  to  be  insufficient  to  have  a  negative 
impact  upon  the  achievement  of  objectives  or  upon  future  development  efforts. 

YELLOW:  Either  (1)  an  approximately  equal  number  of  MOPs  were  rated  as  Positive  Findings 
and  Remaining  Challenges,  OR  (2)  there  are  any  proportion  of  Positive  Findings  with  one  or 
more  Remaining  Challenges  that  are  judged  to  have  limited  the  achievement  of  objectives,  or  to 
have  a  moderate  negative  impact  upon  future  development  efforts.  These  Challenges  should  be 
the  focus  of  development  efforts. 

Red:  Either  (1)  a  preponderance  of  the  MOPs  were  rated  as  Remaining  Challenges  rather  than  as 
Positive  Findings,  OR  (2)  there  are  any  proportion  of  Positive  Findings  with  one  or  more 
Remaining  Challenges  that  are  judged  to  have  prevented  the  achievement  of  objectives,  or  to 
have  a  major  negative  impact  upon  future  development  efforts.  These  Challenges  must  be 
resolved  before  the  continuation  of  development  efforts. 


40 


Joint  Military  Utility  Assessment 


The  scale  used  was  as  follows: 


© 

Green 

A 

Yellow 

# 

Red 

Meets  requirements  with  minor  modifications. 
May  meet  requirements  with  significant  revisions. 
Does  not  meet  requirements. 


In  addition,  the  following  symbol  was  applied  if  it  was  not  possible  to  make  an  evaluative  rating: 


This  requirement  was  not  addressed  by  CINC  21  or 
was  not  addressed  sufficiently  to  allow  for  assessment. 


CINC  21  Product  Technology  Categories 


A  complete  list  of  CINC  2 1  products  along  with  descriptions  is  provided  in  the  What  is  CINC 
21?  section  of  this  report.  For  assessment  purposes,  these  products  have  been  reorganized  into  a 
series  of  technology  categories  that  reflect  the  functional  capabilities  presented  to  CINC  21 
users.  That  is,  the  CINC  21  products  with  which  users  would  interact  directly  have  been 
clustered  into  categories  that  roughly  correspond  with  the  capabilities  that  they  provide. 

The  CINC  21  products  that  are  listed  under  the  Concepts  heading  were  not  assessed.  In  general, 
concepts  do  not  lend  themselves  to  assessment.  Usually,  concepts  have  to  be  assessed  via  the 
specific  technologies  designed  and  developed  to  implement  them.  It  is  only  after  incremental 
successes  that  evidence  for  or  against  the  concept  emerges.  With  ACTDs,  the  concepts  generally 
continue  to  evolve  over  the  life  of  the  program.  Thus,  a  formal  military  utility  assessment  is  an 
inappropriate  method  to  evaluate  concepts.  In  addition,  the  CINC  2 1  products  that  are  strictly 
infrastructure  necessary  to  use  other  products  were  generally  not  included  in  the  assessment. 
Infrastructures  are  the  means  to  an  end.  In  general,  most  technologies  require  some  infrastructure 
before  they  can  be  implemented,  e.g.,  it  would  be  difficult  to  evaluate  a  high-speed  graphics 
capability  without  first  ensuring  that  the  platfonn  and  existing  software  are  sufficiently  upgraded 
to  support  the  new  capability. 

Decision-Focused  Command  and  Control  (DFC2) 

Decision-Focused  Command  and  Control  (DFC2)  is  a  set  of  applications  that  facilitate 
management  of  organizational,  information,  process,  and  resource  functions.  DFC2  applications 
are  characterized  as  three  inter-related  layers:  Presentation,  Business,  and  Data. 

Tsee  description] 

DFC2  products  were  in  an  early  stage  of  development  during  the  JMUA.  As  such,  only  a  few  of 
the  applications  were  directly  assessed.  To  the  extent  that  these  applications  were  dependent  on 
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various  components  of  the  other  layers,  it  is  appropriate  to  assume  that  the  assessment  results  are 
applicable  to  those  applications,  as  well.  Specific  data  were  collected  on  the  following: 

•  Decision  Space  Management  lsee  resultsl 

•  Decision  Summary  [see  resultsl 

•  An  information  management  utility,  Chat  (embedded)  rsee  resultsl 

•  A  time-oriented  Presentation  Viewer,  Battle  Rhythm  rsee  resultsl 

•  A  business  logic  utility,  Status  Rule  Management  rsee  resultsl 

Operational  Packages 

The  Operational  Packages  are  CINC  21 -developed  applications  that  use  the  underlying  enterprise 
architecture.  These  applications  were  categorized  according  to  related  capabilities  and  include: 

Common  Operational  Picture/Geo-Spatial  Visualization 

This  category  is  comprised  of  two  independent  CINC  21  products.  Both  products  were  assessed 
in  separate  venues. 

•  Area  of  Responsibility  Basing  (AORB)  rsee  results!  rsee  description! 

•  Fused  Battlespace  View  (FBV)  rsee  results!  rsee  description! 

The  products  in  this  technology  category  have  the  common  objective  of  providing  operational 
decision-makers  and  supporting  staff  improved  situation  awareness  and  understanding  through 
manipulation  of  graphics  of  information  relevant  to  the  area  of  operations. 

Briefing/Information  Summary 

Several  CINC  2 1  products  provide  a  means  for  operational  personnel  to  prepare  and  disseminate 
summarized  information,  which  is  used  to  promote  situation  awareness  and  comparison  of 
decision  options. 

•  Consequence  Management  Automated  Brief  rsee  resultsl  [see  description! 

•  Course  of  Action  (COA)  Matrix  rsee  resultsl  rsee  description! 

Status  Tracking  and  Linking 

Three  independent  CINC  2 1  products  were  developed  and  assessed  that  assisted  operational 
personnel  working  in  a  distributed,  collaborative  environment  to  organize  and  monitor  task- 
related  information. 

•  Message  Tracker  rsee  results!  rsee  description! 

•  Request  for  Information  (RFI)  Manager  Isee  results]  rsee  description! 

•  TeamApp  Isee  resultsl  Isee  description! 

Time-Based  Event  Management 

Two  independent  CINC  2 1  products  were  developed  to  improve  distributed  staff  awareness  of 
pending  tasks  and  schedules.  These  products  permitted  users  to  manipulate  and  display  task- 
related  information  chronologically. 

•  Task  Management  System  (TMS)  Isee  resultsl  Isee  description! 
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•  Master  Calendar  rsee  resultsl  rsee  description! 

JWID  Coalition  Interoperability  Service  (COINS) 

The  Operational  Package,  JWID  COINS,  is  a  knowledge  management  tool  that  was  assessable 
but  was  not  part  of  the  infrastructure  of  other  CINC  21  products.  In  general,  Knowledge 
Management  is  the  set  of  core  services  that  includes  software  servers  and  applications  for 
information  manipulation  and  information  persistence.  Most  CINC  2 1  products  in  this  category 
are  the  services  that  are  a  result  of  the  infrastructure  applications,  rsee  resultsl  rsee  description! 

Displays 

The  CINC  21  displays  consisted  of  group  and  workstation  products. 

Group  Displays 

This  category  is  comprised  of  two  independent  group  display  products.  These  technologies  are 
intended  to  allow  co-located  personnel  to  reach  common  understanding  of  an  operational 
situation  through  viewing  shared  infonnation  on  a  large  screen. 

•  USPACOM  Video  Wall  (standard-resolution  large  screen  display) 

rsee  resultsl  rsee  description! 

•  USSTRATCOM  Knowledge  Wall  (high-resolution  large  screen  display)  [see  resultsl 
rsee  description! 

Workstation  Displays 

Several  workstation  multi-screen  display  configurations  were  examined  as  CINC  21  products. 
These  products  were  intended  to  increase  the  efficiency  of  work  efforts  via  concurrent  display  of 
task-relevant  information.  Multiple  screens  could  enable  users  to  have  multiple  active 
applications  and  to  manipulate  data  in  one  application  while  monitoring  others. 

•  Multiple  side-by-side  commercial  flat  panel  monitors  [see  resultsl  rsee  description! 

•  PV290  multi-screen  desktop  display  (integrated  three-screen  console)  rsee  results] 
rsee  description! 

Collaboration 

Several  collaboration  products  were  explored  as  part  of  CINC  21.  These  products  were  intended 
to  allow  distributed  operational  personnel  to  synchronize  their  battle  rhythm  and  perform 
distributed  decision-making,  rsee  results!  rsee  description] 

•  DCTS  and  its  component  technologies 

•  Information  Work  Space  (IWS) 

Network  Operations  Tools  and  Services 

The  network  operations  use  a  series  of  applications  to  provide  a  Theater  C4ISR  Coordination 
Center  (TCCC)  display  for  viewing  the  operational  status  of  networks.  Products  assessed  include 
the  following: 

•  Theater  C4ISR  Coordination  Center  Status  Grid  rsee  resultsl  rsee  description! 

•  Network  Operations  Status  Monitoring,  including:  rsee  results!  rsee  description! 


43 


Joint  Military  Utility  Assessment 


o  Remedy  Trouble  Ticket  Workflow  (from  which  is  derived  the  TCCC  Status  Grid) 
o  EICS  tool 

o  Events  Priority  Chart  tool 
o  Link  Analysis  tool 
o  One-Way  Proxy  tool 

•  BMC  Patrol  I  see  results!  [see  description! 

The  Network  Operations  Capability  brief  is  still  under  development  and  was  not  assessed. 

Special  Projects 

The  special  projects  were  single  attempts  to  develop  a  tool  or  application  for  a  specific  use.  In 
some  cases,  they  were  hardware  or  software  applications  that  supported  the  enterprise 
infrastructure  for  a  particular  phase  of  the  ACTD,  but  they  were  unnecessary  or  were  replaced  as 
the  ACTD  concepts  matured.  Most  of  these  projects  involved  hardware  or  software  that  did  not 
lend  themselves  to  a  meaningful  assessment  other  than  to  verify  proper  installation  and 
operation.  The  special  projects  that  were  assessed,  include: 

•  USSTRATCOM  Portal  [see  results!  rsee  description! 

•  JWID  Administrative  Portal  [see  results!  [see  description! 

•  QoS  [see  results!  [see  description! 

•  VPN  [see  results]  [see  description! 

•  Time-Step  VPN 

•  RRAS  VPNs  with  PKI 

•  3Com®  Embedded  Firewall  (EFW)  NIC  Cards  fsee  results!  [see  description] 

The  other  special  projects  were  single  applications  of  tools  and  techniques  to  address  unique 
situations.  None  were  assessed. 


Assessment  of  CINC  21  Products  and  Technology 
Categories 


Decision-Focused  Command  and  Control  (DFC2) 

The  DFC2  technology  category  contains  the  assessments  of  presentation  layer  components, 
decision  space  management,  decision  summary,  embedded  chat,  battle  rhythm,  and  status  rule 
management.8 


:  An  assessment  of  the  TCCC  display  that  was  a  read-only  web  part  on  the  DFC2  is  located  in  the  section,  “Network 
Operations  Tools  and  Services.” 
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Decision  Space  Management 
Assessment  Events 

•  Assessment  of  Initial  Decision  Space  Management  Features  (Nov  02) 

(Appendix  D). 

•  Assessment  of  Status,  Collaboration,  NetOps,  Decision  Summary  and  Battle 

Rhythm  Features  (Nov  02)  (Appendix  E) 

•  Shadow  Play  at  the  Terminal  Fury  ’03  (TF03)  Exercise  (Dec  02)  (Appendix  F) 

•  Human-system  interface  analysis  by  human  factors  professionals  (Appendix  G) 

Demonstration/training  and  hand-on  usage  of  the  technology  were  components  of  the 

sessions.  User  feedback  and  observation  by  assessors  were  the  primary  data  collection 

methods  used.  In  addition,  human  factors  professionals  performed  human-system 

interface  analyses  in  a  laboratory  environment. 

Suitability 

Positive  Findings 

E3.  Synchronization  of  information  management  among  key  players  (e.g.,  joint, 
coalition,  inter-agency,  and  non-governmental  organization  partners) 
increased. 

E4.  Common  understanding  of  plan  progress  improved. 

F2.  The  sharing  of  inputs  among  the  decision-makers  increased. 

HI .  The  currency  of  information  was  sufficient  for  the  decision-makers' 
requirements. 

H2.  The  completeness  of  the  information  was  sufficient  for  decision-makers' 
requirements. 

H3.  The  precision  of  the  information  was  sufficient  for  decision-makers' 
requirements. 

H6.  The  amount  of  relevant  data  available  within  the  decision  cycle  was 
increased. 

18.  CINC  2 1  technologies  were  accepted  by  the  target  users. 

Remaining  Challenges 

Al.  The  accuracy  of  situation  awareness  and  understanding  among  decision¬ 
makers  was  not  increased. 

B 1 .  The  time  needed  to  collect,  identify,  and  integrate  information  was  not 
reduced. 

1 1 .  Staffs  skills  and  knowledge  were  not  better  used 

14.  Outcomes  of  distributed  work  processes  were  not  satisfactory  to  the 
decision-makers. 
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Usability 

Positive  Findings 

D 1 .  Distributed  members  were  able  to  exchange  information  quicker. 

12.  Duplication  of  infonnation  and  work  was  reduced. 

J2.  Conveyance  of  information  on  the  display  device  platforms  was  supported. 

LI .  Time  was  reduced  in  establishing  a  collaborative  framework  (display  of  data 
for  collaborative  efforts). 

Remaining  Challenges 

A2.  Ability  to  scale  and  tailor  visualization  displays  was  not  increased. 

D2.  The  number  of  steps  or  procedures  used  to  exchange  information  were  not 
reduced. 

J3.  The  infonnation  was  not  presented  in  a  form  that  is  usable  by  all  distributed 
members. 

13 .  W orkload  was  not  reduced. 

15.  The  number  of  staff  needed  to  perform  the  task  was  not  reduced. 

16.  CINC  21  technologies  did  not  facilitate  mission-relevant  training. 

Technical 

Positive  Findings 

J 1 .  Information  transmittal  attempts  were  successful. 

Ml .  Server  compatibility  was  not  a  problem. 

M2.  New  applications  and  services  operated  within  existing  applications  and 
services. 

Remaining  Challenges 

A3.  Decision  support  and  knowledge  management  was  not  advanced. 

General  Findings 

Positive  Findings 

•  Users  endorsed  the  integrated  view  of  the  decision  space  as  valuable  to  attain 
and  maintain  situation  awareness. 

•  The  Decision  Space  Management  is  helpful  in  monitoring  decision  points  and 
CCIRs. 

•  Detail  and  analyses  available  on  drill-down  is  an  efficient  mechanism  for 
exchange  and  presentation  of  infonnation. 
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Remaining  Challenges 

•  The  presentation  format  interferes  with  users’  ability  to  navigate  efficiently 
within  the  decision  space. 

•  The  number  of  levels  of  relationships  that  are  displayed  for  operations  is 
inadequate. 

•  The  functionality  for  creating  and  retrieving  linked  products  is  too  limited. 

•  The  refresh  rate  is  too  slow. 

•  The  inability  to  save  a  configuration  adds  considerably  to  the  user’s  workload. 

•  Access  to  applications  is  not  stable. 

Decision  Summary 

Assessment  Events 

•  Assessment  of  Status,  Collaboration,  NetOps,  Decision  Summary,  and  Battle 
Rhythm  Features  (Nov  02)  (Appendix  E) 

•  Shadow  Play  at  the  Terminal  Fury  ’03  (TF03)  Exercise  (Dec  02)  (Appendix  F) 

Data  were  collected  from  operational  users  during  two  events.  Demonstration/training 
and  hand-on  usage  of  the  technology  were  components  of  the  sessions.  User  feedback 
and  observation  by  assessors  were  the  primary  data  collection  methods  used. 

Suitability 

Positive  Findings 

F2.  The  sharing  of  inputs  among  the  decision-makers  increased. 

Remaining  Challenges 

18.  Target  users  had  reservations  about  the  utility  of  the  technology. 

Usability 

Positive  Findings 

-NONE-- 

Remaining  Challenges 

A2.  Ability  to  scale  and  tailor  visualization  displays  was  not  increased. 

Technical 

Positive  Findings 

-NONE- 

Remaining  Challenges 

-NONE- 
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General  Findings 

Positive  Findings 

•  Users  endorsed  the  concept  of  a  Decision  Summary  to  provide  decision¬ 
makers  with  relevant  background  information  and  to  function  as  a  briefing 
tool. 

Remaining  Challenges 

•  A  mechanism  is  needed  to  display  relationships  between  decision  points  and 
the  ongoing  decision  development  process. 

•  The  display  should  have  the  capability  to  stand  alone  as  a  briefing  tool. 


Chat 


Assessment  Events 

•  Assessment  of  Status,  Collaboration,  NetOps,  Decision  Summary,  and  Battle 
Rhythm  Features  (Nov  02)  (Appendix  E) 

•  Shadow  Play  at  the  Terminal  Fury  ’03  (TF03)  Exercise  (Dec  02)  (Appendix  F) 

Data  were  collected  from  operational  users  during  two  events.  Demonstration/training 
and  hand-on  usage  of  the  technology  were  components  of  the  sessions.  User  feedback 
and  observation  by  assessors  were  the  primary  data  collection  methods  used. 

Suitability 

Positive  Findings 

E3.  Synchronization  of  information  management  among  key  players  (e.g.,  joint, 
coalition,  inter-agency,  and  non-governmental  organizations)  increased. 

F2.  The  sharing  of  inputs  among  the  decision-makers  increased. 

Remaining  Challenges 

18.  Target  users  had  reservations  about  the  utility  of  the  technology. 

Usability 

Positive  Findings 

A2.  Ability  to  scale  and  tailor  visualization  displays  increased. 

El.  The  number  of  actions  needed  to  coordinate  between  participants  was 
reduced. 

16.  CINC  21  technologies  facilitated  mission-relevant  training. 

Remaining  Challenges 

J2.  Conveyance  of  information  on  the  display  device  platforms  was  not 
supported. 
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Technical 

Positive  Findings 

J1 .  Information  transmittal  attempts  were  successful. 

Ml .  Server  compatibility  was  not  a  problem. 

Remaining  Challenges 

A3.  Decision  support  and  knowledge  management  was  not  advanced. 

General  Findings 

Positive  Findings 

•  Object-specific  chat  sessions  are  easy  to  use. 

•  The  ability  to  “tear  off’  and  retain  multiple  active  chats  received  high  ratings. 
Remaining  Challenges 

•  Time  stamping  on  messages  is  needed. 

•  There  is  no  alerting  system  for  messages  received. 

•  Users  need  the  ability  to  resize  chat  windows. 

•  Individuals  who  are  not  currently  logged  on  to  DFC2  cannot  be  included  in 
object-specific  chat. 

Battle  Rhythm 

Assessment  Events 

•  Assessment  of  Status,  Collaboration,  NetOps,  Decision  Summary,  and  Battle 
Rhythm  Features  (Nov  02)  (Appendix  E) 

•  Shadow  Play  at  the  Terminal  Fury  ’03  (TF03)  Exercise  (Dec  02)  (Appendix  F) 

Data  were  collected  from  operational  users  during  two  events.  Demonstration/training 
and  hand-on  usage  of  the  technology  were  components  of  the  sessions.  User  feedback 
and  observation  by  assessors  were  the  primary  data  collection  methods  used. 

Suitability 

Positive  Findings 

F2.  The  sharing  of  inputs  among  the  decision-makers  increased. 

Remaining  Challenges 

H7.  Faster  synchronization  of  battle  rhythms  was  not  achieved. 

18.  Target  users  had  reservations  about  the  utility  of  the  technology. 

Usability 


Positive  Findings 

-NONE- 
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Remaining  Challenges 

A2.  Ability  to  scale  and  tailor  visualization  displays  was  not  increased. 

Technical 

Positive  Findings 

-NONE- 

Remaining  Challenges 

-NONE- 

General  Findings 

Positive  Findings 

•  Users  endorsed  the  concept  of  a  Battle  Rhythm  as  central  to  the  DFC2  concept 
of  operations. 

Remaining  Challenges 

•  There  is  insufficient  fidelity  to  allow  for  filtering  and  display  of  specific 
activities. 

•  The  tool  must  be  able  to  display  links  between  activities. 

•  Users  need  the  ability  to  display  multiple  battle  rhythms  concurrently. 

Status  Rule  Management 

Assessment  Events 

•  Assessment  of  Status,  Collaboration,  NetOps,  Decision  Summary,  and  Battle 
Rhythm  Features  (Nov  02)  (Appendix  E) 

•  Shadow  Play  at  the  Terminal  Fury  03  (TF03)  Exercise  (Dec  02)  (  Appendix  F) 

Status  Rule  Management  capabilities  are  provided  in  conjunction  with  the  Decision 
Space  Management  component  of  DFC2.  Data  were  collected  from  operational  users 
during  two  events.  Demonstration/training  and  hand-on  usage  of  the  technology  were 
part  of  the  sessions.  User  feedback  and  observation  by  assessors  were  the  primary 
data  collection  methods  used.  In  addition,  human  factors  professionals  performed  a 
heuristic  evaluation  of  its  usability  in  a  laboratory  environment. 

Suitability 

Positive  Findings 

Al.  The  accuracy  of  situation  awareness  and  understanding  among  decision¬ 
makers  increased. 

F2.  The  sharing  of  inputs  among  the  decision-makers  increased. 

18.  CINC  2 1  technologies  were  accepted  by  the  target  users. 

Remaining  Challenges 

-NONE- 
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Usability 

Positive  Findings 

-NONE- 

Remaining  Challenges 

A2.  Scalability  and  the  ability  to  tailor  visualization  displays  was  not  increased. 

12.  Duplication  of  information  and  work  was  not  reduced. 

13.  Workload  was  not  reduced. 

16.  CINC  21  technologies  did  not  facilitate  mission-relevant  training. 

J2.  Conveyance  of  information  on  the  display  device  platforms  was  not 
supported. 

Technical 

Positive  Findings 

Jl.  Information  transmittal  attempts  were  successful. 

Ml .  Server  compatibility  was  not  a  problem. 

M2.  New  applications  and  services  operated  within  existing  applications  and 
services. 

Remaining  Challenges 

A3.  Decision  support  and  knowledge  management  was  not  advanced. 

General  Findings 

Positive  Findings 

•  Display  of  status  attributes  is  predicted  to  increase  situation  awareness. 

•  Users  endorsed  the  concept  of  displaying  status  attributes  of  decision  points 
and  CCIRs. 

Remaining  Challenges 

•  Status  attributes  and  meanings  are  not  standardized. 

•  Training  and  guidelines  for  use  are  needed. 

•  The  refresh  rate  and  the  ability  to  save  status  parameters  must  be  improved. 

•  The  status  display  must  be  personalized  so  that  users  with  different 
information  needs  can  choose  the  level  of  detail  to  view. 

•  The  status  rule  entry  and  editing  processes  are  very  cumbersome  and  time- 
consuming. 

DFC2  Summary 

Users  rated  the  integrated  view  of  the  decision  space  provided  by  the  Decision  Space 
Management  component  of  DFC2  as  valuable.  They  estimate  that  it  will  help  them  to 
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maintain  situation  awareness  of  multiple  events.  They  also  felt  it  will  be  helpful  to  monitor 
specific  decision  points  and  CCIRs.  The  ability  to  drill  down  from  the  top-level  display  for 
more  detailed  information  was  judged  to  be  an  effective  way  to  provide  easy  access  to 
complete,  relevant  information.  There  are  many  usability  concerns,  however.  The 
presentation  format  and  users’  ability  to  navigate  must  be  improved.  In  addition,  capabilities 
for  creating  and  retrieving  linked  products  must  be  improved. 

When  users  saw  a  static  display  of  the  Decision  Summary,  they  endorsed  the  concept  of  a 
feature  that  would  provide  a  detailed  synopsis  of  analyses,  recommendations,  and  rationale 
about  a  decision  point  or  other  decision  object.  They  judged  that  this  type  of  feature  could 
serve  a  valuable  function  as  a  briefing  tool.  However,  when  the  Decision  Summary  was 
interactive,  users  provided  very  low  ratings.  The  current  implementation  in  DFC2  is  not 
adequate  to  provide  a  decision-maker  within  the  command  with  instant  and  comprehensive 
situation  awareness.  For  example,  there  is  not  a  clear  relationship  to  decision  points  and  the 
ongoing  decision  development  process.  DFC2  was  assessed  as  incapable  of  standing  alone  as 
a  briefing  tool  for  a  senior  decision-maker. 

When  users  saw  a  static  display  of  the  Battle  Rhythm,  they  endorsed  the  concept  of  the 
feature  as  central  to  the  concept  of  operations  of  DFC2.  They  were  enthusiastic  about  the 
ability  to  display  and  compare  multiple  battle  rhythms  of  multiple  groups  and  to  filter  a  battle 
rhythm  display  to  show  specified  categories  of  activities.  However,  when  the  Battle  Rhythm 
was  interactive,  users  provided  very  low  ratings.  The  current  implementation  in  DFC2  does 
not  provide  adequate  fidelity.  Users  judged  it  as  not  fully  operative  and  as  the  least  mature  of 
the  DFC2  technologies. 

Users  reported  that  most  collaboration  within  the  JOC  is  currently  face-to-face.  However, 
they  could  see  high  potential  value  of  the  DFC2  embedded  Chat  feature,  especially  if 
collaborative  document  editing  was  possible.  DFC2  object-specific  chat  is  easy  to  use,  but 
lacks  time  stamp  and  alerting  features.  At  present,  user-selected  views  are  not  retained  nor 
can  the  chat  windows  be  resized.  Users  can  now  access  NetMeeting  to  contact  individuals 
who  are  not  logged  on  to  DFC2;  however,  the  capability  to  include  these  individuals  in  the 
chat  sessions  was  requested. 

Users  saw  value  in  the  ability  provided  by  the  DFC2  Status  Rule  Management  to  display 
status  attributes  of  decision  points  and  CCIRs.  There  was  not  consensus,  however,  about  how 
much  detail  should  be  displayed.  It  was  recognized  that  personalization  of  this  feature  is 
needed.  Also  needed  are  standardization  of  status  attributes  and  their  meanings.  Guidelines 
for  use  are  essential.  The  user  must  interact  with  numerous  dialog  boxes  during  several  steps 
to  enter  or  edit  status  attributes.  This  unwieldy  procedure  caused  heavy  workload  and  user 
dissatisfaction — the  current  Status  Rule  Management  feature  was  ruled  unacceptable  because 
of  this.  The  placement  and  naming  of  menu  bars  caused  confusion;  in  addition,  selecting 
pull-down  menu  items  required  precise  mouse  contact.  The  legibility  and  organization  of  the 
input  and  display  fields  was  poor. 

The  concept  of  DFC2 — applications  that  facilitate  the  definition,  visualization,  analysis,  and 
management  of  decision-making  activities  associated  with  joint  command  and  control — was 
well  received  by  users.  However,  there  is  currently  a  lack  of  functioning  links  between 
component  parts  and  between  decision  objects  within  component  parts.  The  document 
linking  and  management  services  in  their  current  stage  of  development  do  not  fulfill  users’ 
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needs.  In  addition,  concerns  regarding  the  impact  of  DFC2  upon  workload  and  organizational 
procedures  remain.  Therefore,  Suitability  is  rated  as  YELLOW. 

Usability  problems  were  experienced  throughout  the  assessment  events,  especially  in  areas  of 
configuration  retention  and  navigation.  Excessive  use  of  scrollbars  limit  users’  ability  to 
maneuver,  and  they  present  a  confusing  display.  Many  functions  require  numerous  steps  as 
well  as  interaction  with  multiple  dialog  boxes.  Also,  the  use  of  screen  real  estate  is  less  than 
optimal;  web  parts  are  often  located  partially  or  completely  off  the  screen.  As  a  result  of 
these  problems,  Usability  is  rated  as  YELLOW. 

A  slow  refresh  rate  and  some  instability  of  component  parts  occurred  throughout  the 
assessment  events.  These  problems  interfered  with  users  as  they  attempted  to  input  or  access 
information.  There  were  intermittent  crashes  of  DFC2  during  TF03;  these  crashes  were 
associated  with  data  losses.  The  Technical  rating  is  YELLOW. 

It  has  to  be  noted  that  DFC2  was  in  the  very  early  stages  of  development  during  the  JMUA. 

In  some  cases,  it  was  only  in  a  prototype  form  that  lacked  the  richness  of  the  overall  concept. 
Thus,  the  JMUA  is  based  on  very  limited  data.  As  the  concept  matures  and  goes  to  full 
development,  a  follow-on  evaluation  can  better  assess  the  overall  utility,  usability,  and  ability 
to  support  the  concept. 


Decision  Focused 

Suitability 

Usability 

Technical 

Command  &  Control 

A 

A 

A 

Operational  Packages 

Common  Operational  Picture/Geo-Spatial  Visualization 

This  technology  category  contains  the  assessments  of  AORB  and  FBV.  Each  technology  is 
examined  with  regard  to  visual/prcscntation  and  content.  Data  regarding  visualization  are 
combined  into  COI-1,  while  information  about  content  is  in  COI-2. 

Area  of  Responsibility  Basing 

Assessment  Events 

•  State  of  the  AORB  system  (Jan  02)  (Appendix  H) 

AORB  feedback  was  collected  from  operational  users  on  the  state  of  the  AORB 
system  (as  of  January  2002)  with  regard  to  the  Critical  Infrastructure  Protection  (CIP) 
and  Component  Real  Property  (CRP).  One  user  of  the  CIP  and  two  users  of  the  CRP 
were  shown  the  AORB  and  asked  to  perform  common  tasks.  Afterwards,  each 
completed  a  questionnaire  that  addressed  various  issues  concerning  functional  utility 
and  interface  usability. 

Suitability 

Positive  Findings 

Al.  The  accuracy  of  situation  awareness  and  understanding  among  decision¬ 
makers  increased. 
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A3.  Decision  support  and  knowledge  management  was  advanced. 

B 1 .  The  time  needed  to  collect,  identify,  and  integrate  information  was  reduced. 

B3.  The  time  to  display  requested  information  decreased,  so  that  additional  time  is 
available  for  analyzing  decision  options. 

Cl.  Planners’  satisfaction  with  the  decision  increased. 

Remaining  Challenges 

C2.  The  degree  to  which  decisions  met  organizational  standards  was  not 
improved. 

Usability 

Positive  Findings 

D 1 .  Distributed  members  were  not  able  to  exchange  information  quicker. 

D2.  The  number  of  steps  or  procedures  used  to  exchange  information  was 
reduced. 

13.  Workload  was  reduced. 

15.  The  number  of  staff  needed  to  perfonn  the  task  was  reduced. 

J3.  The  information  was  presented  in  a  fonn  that  was  usable  by  all  distributed 
members. 

Remaining  Challenges 

-NONE- 

Technical 

Positive  Findings 

-NONE- 

Remaining  Challenges 

H2.  The  completeness  of  the  infonnation  was  not  sufficient  for  decision-makers’ 
requirements. 

General  Findings 

Positive  Findings 

•  The  tool  provided  a  significant  reduction  in  task  time  (from  many  hours  to 
minutes). 

•  Fewer  personnel  were  required  to  meet  mission  needs  when  using  the  tool. 

•  It  was  considered  easy  to  learn  and  use. 

•  The  tool  enabled  operators  to  visualize  geo-spatial  planning  parameters. 

•  Users  found  the  technology  an  improvement  over  current  systems. 
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Remaining  Challenges 

•  The  tool  lacks  access  to  necessary  operational  data  bases. 

•  Use  of  the  tools  requires  a  specialized  web  application. 

AORB  Summary 

Suitability  is  rated  as  Green  since  AORB  provided  useful  capabilities  and  improved 
efficiency.  Users  responded  that  this  technology  would  help  them  meet  their  operational 
requirements.  They  were  also  able  to  navigate  using  the  technology  with  minimal 
instruction.  However,  there  are  still  many  changes  that  should  be  made  to  the  user 
interfaces  to  improve  its  usability.  They  are  not  issues  that  prevent  users  from  ultimately 
accomplishing  their  tasks.  Usability  is,  therefore,  rated  as  Green.  Unfortunately, 
incomplete  data  sources  behind  AORB  prevent  it  from  being  used  in  a  real  operational 
setting.  This  major  hurdle  is  responsible  for  the  Technical  rating  of  YELLOW. 


Area  of  Responsibility 

Suitability 

Usability 

Technical 

A 

Basing 

© 

© 

A 

Fused  Battlespace  View 

Assessment  Events 

•  USSTRATCOM  Consequence  Management/Response  Demonstration  (Oct  02) 
(Appendix  I) 

The  FBV  was  part  of  the  USSTRATCOM  demonstration  event  in  October  2002. 
Senior  decision-makers  where  guided  through  an  exercise,  and  then  feedback  on 
how  the  CINC  21  technologies  (including  FBV)  were  perceived  to  support  and 
increment  work  processes  in  Consequence  Management/Response.  In  addition, 
assessment  team  members  had  an  opportunity  to  interview  potential  Action 
Officer  (AO)  users  and  to  comment  on  the  user  interface. 

Suitability 

Positive  Findings 

B 1 .  The  time  needed  to  collect,  identify,  and  integrate  information  was  reduced. 

B3.  The  time  to  display  requested  information  decreased,  so  that  additional  time  is 
available  for  analyzing  decision  options. 

Cl.  Planner’s  satisfaction  with  the  decision  option  increased. 

HI.  The  currency  of  information  was  sufficient  for  the  decision-maker’s 
requirements. 

H3.  The  precision  of  information  was  sufficient  for  decision-makers 
requirements. 
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H6.  The  amount  of  relevant  data  available  within  the  decision  cycle  was 
increased. 

18.  CINC  21  technologies  were  accepted  by  the  target  users. 

Remaining  Challenges 

Al.  The  accuracy  of  situation  awareness  and  understanding  among  decision¬ 
makers  did  not  increase. 

H2.  The  completeness  of  the  information  was  not  sufficient  for  decision-makers' 
requirements. 

Usability 

Positive  Findings 

A2.  Ability  to  scale  and  tailor  visualization  displays  increased. 

D2.  The  number  of  steps  or  procedures  used  to  exchange  information  was 
reduced. 

El .  The  number  of  actions  needed  to  coordinate  between  participants  was 
reduced. 

13 .  W orkload  was  reduced. 

15.  The  number  of  staff  needed  to  perform  the  task  reduced 

16.  CINC  21  technologies  facilitated  mission-relevant  training 

17.  Watchstanders  were  able  to  relay  information  faster  when  managing  routine 
theater  activity. 

LI .  The  time  was  reduced  in  establishing  a  collaborative  framework. 

Remaining  Challenges 

J2.  Conveyance  of  information  on  the  display  device  platforms  was  not 
supported. 

J3.  The  information  was  not  presented  in  a  form  that  is  usable  by  all  distributed 
members. 


Technical 

Positive  Findings 

A3.  Decision  support  and  knowledge  management  was  advanced. 
Remaining  Challenges 

K5.  The  system  did  not  provide  a  secure  means  for  exchanging  information 
among  distributed  users. 

General  Findings 

Positive  Findings 

•  The  tool  provides  current  information  to  all  users. 
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•  It  decreases  the  number  of  steps  and  staff  needed  to  develop  and  exchange 
information. 

•  Users  found  it  to  be  easy  to  understand  the  content  and  to  navigate  through  the 
workflow. 

•  Geo-spatial  displays  with  drill-down  capability  aided  situation  awareness. 

•  Target  users  accepted  the  technology. 

Remaining  Challenges 

•  Insufficient  drill-down  capability  was  available. 

•  Users  were  unable  to  exchange  information  across  security  levels. 

•  Some  user  interface  improvements  are  needed  (font  size,  link  colors,  etc.). 

•  Procedural  issues  regarding  dynamic  updating  need  to  be  resolved. 

FBV  Summary 

The  FBV  received  high  marks  for  Suitability  as  users  were  satisfied  that  their  operational 
issues  were  addressed.  When  compared  to  how  things  are  done  today,  the  FBV  could 
help  users  find  infonnation  faster  and  with  fewer  personnel  involved.  Suitability  is, 
therefore,  rated  as  Green.  Usability  also  received  a  Green  rating  because  users  were  able 
to  navigate  the  interface  with  very  little  instruction.  At  the  time  of  the  assessment, 
however,  users  wanted  to  access  classified  information,  which  the  FBV  was  unable  to 
provide.  This  lack  of  access  to  classified  infonnation  results  in  a  Yellow  rating  for 
Technical. 


Suitability 

Usability 

Technical 

Fused  Battlespace  View 

© 

© 

A 

Briefing/Information  Summary 

This  technology  category  contains  the  assessments  for  the  Consequence  Management  Automated 
Brief  and  the  CO  A  Matrix. 

Consequence  Management  Automated  Brief 

Consequence  Management  Automated  Brief  is  a  web-based  briefing  and  information 
management  tool  that  can  be  used  to  capture,  archive,  and  organize  text  and  graphics  for 
presentation.  It  is  intended  to  enhance  the  presentation  of  information  by  improving  the 
display  and  access  to  consequence  management  related  infonnation. 

Assessment  Events 

•  Heuristic  evaluation  of  interface  features  (Appendix  J) 

•  LAO  with  civilian  participants  (Jul  02)  (  Appendix  K) 

The  version  of  the  Consequence  Management  tool  evaluated  in  July  2002  was  a  web- 
based  prototype  intended  to  be  a  component  of  the  Rapid  Force  Employment  portal 
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developed  for  USPACOM  by  CINC  21.  The  usability  focus  was  on  viewing  existing 
information  rather  than  on  generating  new  presentations.  The  two  general  goals  were 
to  measure  (1)  processing  time  as  an  indicator  of  workload  and  (2)  usability  and 
workload  as  perceived  by  the  users.  Human-computer  interface  evaluation  helped 
identify  the  factors  that  affected  processing  time,  usability,  and  workload. 

Suitability 

Positive  Findings 

II.  In  general,  the  staffs  skills  and  knowledge  were  better  utilized. 

Remaining  Challenges 

-NONE- 

Usability 

Positive  Findings 

J3.  The  information  was  presented  in  a  fonn  that  is  usable  by  all  distributed 
members  (except  for  the  two  home  pages  which  were  overly  long  and 
complex). 

Remaining  Challenges 

13.  Workload  was  not  always  kept  to  a  minimum. 

Technical 

Positive  Findings 

-NONE- 

Remaining  Challenges 

A3.  Decision  support  and  knowledge  management  was  not  advanced. 

General  Findings 

Positive  Findings 

•  The  tool  provides  a  well-implemented  structure  for  capturing  information  and 
for  creating,  managing,  retrieving,  and  displaying  presentations. 

•  Most  pages  (with  the  exception  of  the  home  pages)  are  clear,  concise, 
comprehensive,  and  organized  into  well-structured  categories. 

Remaining  Challenges 

•  Lengthy,  complex  home  pages  complicate  navigation  and  information 
retrieval  and  impose  excessive  memory  demands. 

•  Numerous  bright  colors  on  home  pages  distract  users  and  hasten  visual 
fatigue. 

•  Map  graphics  demand  high  bandwidth,  resulting  in  long  download  times 
during  peak  periods. 
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•  Improved  error  detection  and  correction,  navigation  aids,  and  online  help  are 
needed. 

Consequence  Management  Automated  Brief  Summary 

Suitability  was  assigned  a  Green  rating  because,  from  the  viewpoint  of  infonnation  access  and 

management,  this  tool 
is  capable  of  creating, 
managing,  retrieving, 
and  displaying 

information,  which  are  the  primary  activities  for  which  it  was  designed.  In  addition,  the  format, 
content,  and  layout  of  the  majority  of  the  Consequence  Management  (CM)  pages  are  good. 
However,  the  two  home  pages  are  too  long  and  contain  too  many  links,  creating  problems  in 
finding  infonnation  and  remembering  where  it  is  located.  Because  home  pages  are  crucial  for  a 
tool’s  usability,  the  CM  tool  was  given  a  YELLOW  Usability  rating. 

The  Technical  aspects  of  this  technology  were  not  fully  assessed,  but  indications  are  that 
response  times  are  very  slow  with  some  of  the  graphics  typically  used  in  briefs. 
Therefore,  Technical  is  given  a  YELLOW  rating. 


Consequence  Management 

Suitability 

Usability 

Technical 

A 

Automated  Brief 

© 

A 

A 

CO  A  Matrix 

Assessment  Events 

•  USSTRATCOM  Consequence  Management/Response  Demonstration  (October 
2002)  (  Appendix  I) 

The  COA  Matrix  was  part  of  the  USSTRATCOM  demonstration  event  in  October 
2002.  Senior  decision-makers  were  guided  through  an  exercise,  and  then  feedback 
was  collected  regarding  how  the  CINC  2 1  technologies  (including  the  COA 
Matrix)  were  perceived  to  support  work  processes  in  Consequence 
Management/Response.  In  addition,  potential  AO  users  commented  on  the  user 
interface  during  post-demonstration  interviews. 

Suitability 

Positive  Findings 

Al.  The  accuracy  of  situation  awareness  and  understanding  among  decision¬ 
makers  was  increased. 

B 1 .  The  time  needed  to  collect,  identify,  and  integrate  information  was  reduced. 

B3.  The  time  to  display  requested  information  decreased,  so  that  additional  time  is 
available  for  analyzing  decision  options. 

C 1 .  The  planners'  satisfaction  with  the  decision  increased. 

E2.  The  synchronization  of  decision  tasks  among  key  players  improved. 

HI.  The  currency  of  information  was  sufficient  for  the  decision-maker’s 
requirements. 

H6.  The  amount  of  relevant  data  available  within  the  decision  cycle  increased. 
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F2.  The  sharing  of  inputs  among  the  decision-makers  increased 
F3.  The  ability  to  effectively  present  CO  As  was  improved. 

F4.  The  ability  to  evaluate  alternative  CO  As  was  improved. 

G2.  Decision  choices  for  crisis  response  were  framed  faster. 

18.  CINC  21  technologies  were  accepted  by  the  target  users. 

Remaining  Challenges 

H2.  The  completeness  of  the  information  was  not  sufficient  for  decision-makers' 
requirements. 

H3.  The  precision  of  information  was  not  sufficient  for  decision-makers 
requirements. 

Usability 

Positive  Findings 

A2.  Ability  to  scale  and  tailor  visualization  displays  increased. 

D2.  The  number  of  steps  or  procedures  used  to  exchange  information  was 
reduced. 

El.  The  number  of  actions  needed  to  coordinate  between  participants  was 
reduced. 

13.  Workload  was  reduced. 

15.  The  number  of  staff  needed  to  perfonn  the  task  was  reduced. 

16.  CINC  21  technologies  facilitated  mission-relevant  training. 

17.  Watchstanders  were  able  to  relay  information  faster  when  managing  routine 
theater  activity. 

J3.  The  information  was  presented  in  a  form  that  was  usable  by  all  distributed 
members. 

LI.  The  time  was  reduced  in  establishing  a  collaborative  framework. 

Remaining  Challenges 

J2.  Conveyance  of  information  on  the  display  device  platforms  was  not  supported. 
Technical 

Positive  Findings 

A3.  The  decision  support  and  knowledge  management  was  advanced. 

Remaining  Challenges 

H2.  The  completeness  of  the  information  was  not  sufficient 

K5.  The  system  did  not  provide  a  secure  means  for  exchanging  information 
among  distributed  users. 
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General  Findings 

Positive  Findings 

•  Top-level  decision  information  was  aggregated  in  an  understandable  form. 

•  CO  A  Matrix  promotes  side-by-side  comparison  of  alternative  CO  As. 

•  The  tool  decreases  the  number  of  steps  and  staff  needed  to  develop  COAs. 

•  Users  reported  a  high  degree  of  satisfaction  with  the  technology. 

Remaining  Challenges 

•  Some  user  interface  improvements  are  needed  (font  size,  colors,  etc.). 

•  Easy  access  to  business  rules  concerning  color  codes  and  terms  is  needed. 

•  Users  need  to  be  able  to  exchange  information  across  security  levels. 

COA  Matrix  Summary 

For  Suitability  and  Usability,  the  COA  Matrix  received  Green  ratings.  While  it  is 
certainly  the  case  that  some  improvements  are  needed  (colors  are  over-saturated,  text  font 
is  too  small  and  has  too  little  contrast  with  background,  access  to  business  rules  needs  to 
be  easier,  etc.),  users  understood  the  color  scheme  and  were  quickly  able  to  rank  the 
various  alternatives  and  select  the  preferred  COA.  Senior  decision-makers  liked  the  side- 
by-side  comparisons  and  found  the  technology  to  be  usable  and  an  asset  for  them.  As 
with  many  of  the  USSTRATCOM  technologies,  the  issue  of  security  and  how  to 
incorporate  classified  and  unclassified  into  the  technology  was  cause  for  concern.  This 
resulted  in  a  Yellow  rating  for  Technical. 


Suitability 

Usability 

Technical 

COA  Matrix 

© 

© 

A 

Status  Tracking  and  Linking 

Status  Tracking  and  Linking  contains  the  assessments  of  Message  Tracker,  RFI  Manager,  and 
Team  App.  All  three  technologies  were  examined  in  terms  of  interface  features  and  training 
requirements. 

Message  Tracker 

The  Message  Tracker  is  a  web-based  document  management  system  for  organizing 
electronic  messages  that  are  sent  between  military  staff  members.  It  enhances  the  processing 
of  orders  by  providing  capabilities  for  capturing,  filtering,  archiving,  and  organizing 
messages.  It  allows  users  to  group  messages  by  operational  categories  and  to  link  supporting 
messages  to  relevant  events  and  CCIRs.  It  is  intended  to  enhance  the  sharing  of  operationally 
relevant  information  and  to  make  task  assignments  based  on  the  messages. 

Assessment  Events 

•  Heuristic  evaluation  of  interface  features  (Appendix  L) 
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•  LAO  at  USPACOM  Joint  Operations  Center  (JOC)  (September  2002)  (Appendix 

Ml 

•  LAO  with  civilian  participants  (July  2002)  (Appendix  N) 

Usability  evaluations  by  human  factors  analysts  supplemented  performance  and 
questionnaire  data  collected  from  six  military  and  five  civilian  participants  to  assess 
the  capabilities  of  the  Message  Tracker. 

Suitability 

Positive  Findings 

Al.  The  accuracy  of  situation  awareness  and  understanding  among  decision¬ 
makers  may  increase. 

F2.  Sharing  of  inputs  among  the  decision-makers  increased. 

H6.  The  amount  of  relevant  data  available  within  the  decision  cycle  was  predicted 
to  increase. 

Remaining  Challenges 

11.  The  user's  skills  and  knowledge  were  not  always  well-utilized. 

18.  Message  Tracker  (in  its  current  form)  was  not  enthusiastically  accepted  by  the 
target  users. 

Usability 

Positive  Findings 

D 1 .  Distributed  members  should  be  able  to  exchange  information  quicker. 

12.  Duplication  of  information  and  work  was  reduced. 

Remaining  Challenges 

13.  Workload  was  not  appreciably  reduced. 

16.  Message  Tracker  did  not  always  facilitate  mission-relevant  training. 

J3.  Information  was  not  always  presented  in  a  form  usable  by  all  distributed 
members  (feedback,  refresh  issues). 

Technical 

Positive  Findings 

M 1 .  Server  compatibility  was  not  a  problem. 

M2.  New  applications  and  services  operated  within  existing  applications  and 
services. 

Remaining  Challenges 

A3.  Decision  support  and  knowledge  management  was  not  advanced. 

General  Findings 

Positive  Findings 
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•  The  tool  links  messages  with  significant  events. 

•  It  deals  effectively  with  military  message  formats. 

•  Most  web  pages  (except  entry  pages)  were  easy  to  understand  and  navigate. 

•  The  tool  enhances  access  to  messages  and  information. 

Remaining  Challenges 

•  Online  help  for  new  users,  and  additional  navigation  tools  for  experienced 
users  are  lacking. 

•  The  information  structure  does  not  facilitate  navigation. 

•  Memory  and  workload  demands  are  too  high. 

•  The  bright  colors  distract  users  and  hasten  visual  fatigue. 

•  The  interface  cannot  be  customized  to  user  tasks. 

•  Alerting,  searching,  display  manipulation,  and  feedback  mechanisms  are 
inadequate. 

•  An  excessive  amount  of  practice  is  currently  needed  to  reach  proficiency. 

•  Refresh  rate  may  be  too  slow  for  operational  use. 

Message  Tracker  Summary 

The  Message  Tracker  is  effective  at  linking  messages  to  significant  events  and  increasing 
staffs  access  to  information,  which  are  all  important  suitability  features.  Therefore, 
Suitability  is  rated  Green.  Improving  interface  features,  error  prevention  and  detection, 
online  help,  and  search  capabilities  can  enhance  usability.  However,  none  of  these 
shortcomings  prevent  users  from  accomplishing  their  tasks,  and  Usability  is  rated  as 
Yellow.  The  Technical  rating  is  Green,  since  a  slow  refresh  rate  was  the  only  technical 
problem  encountered.  However,  use  in  an  actual  operational  environment  is  needed  for  an 
ample  assessment  of  technical  capabilities. 


Suitability 

Usability 

Technical 

Message  Tracker 

© 

A 

© 

RFI  Manager 

The  RFI  Manager  is  a  web-based  document  management  system  for  handling  formal  requests 
for  information  within  and  between  military  staffs.  Currently,  this  is  a  CINC  21  stand-alone 
application.  It  is  not  linked  to  the  knowledge  core  at  this  time.  It  is  intended  to  enhance  the 
processing  of  RFIs  by  improving  the  display  of  RFIs  and  the  management  of  the  RFI 
database.  A  component  of  the  RFI  Manager  system  is  information  management  via  tasking  of 
registered  users  by  the  manager.  At  USPACOM  headquarters,  it  uses  local  databases  and 
web  forms  that  link  to  the  local  database.  This  combination  allows  for  a  local  capability  to 
track  requests  and  identify  due  dates,  status,  and  responsible  individual/organization.  The 
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tool  has  been  used  extensively  at  the  Joint  Task  Force  headquarters  by  III  MEF  and 
COMSEVENTHFLT. 

Assessment  Events 

•  LAO  at  USPACOM  JOC  (September  2002)  (Appendix  O) 

•  LAO  with  civilian  participants  (July  2002)  (Appendix  P) 

•  Special  assessments  by  III  MEF  in  Cobra  Gold  02  and  in  Operation  Bevel 
Express  02. 

During  September  2002,  six  individuals  in  the  JOC  at  USPACOM  received  training 
on  the  RFI  Manager  and  subsequently  completed  a  series  of  tasks  using  it. 
Assessment  measures  consisted  of  direct  observation  of  performance  as  well  as  user 
responses  to  survey  and  interview  items.  A  similar  data  collection  effort  was  also 
conducted  at  the  Space  and  Naval  Warfare  Systems  Center,  San  Diego  (SSC  San 
Diego)  with  five  civilian  participants.  A  heuristic  evaluation  by  human  factors 
professionals  that  examined  the  usability  characteristics  of  the  RFI  Manager  was 
included  in  the  July  2002  assessment.  Ill  MEF  and  Commander  Seventh  Fleet 
(COMSEVENTHFLT)  have  used  it  in  operational  exercises. 

Suitability 

Positive  Findings 

F2.  The  sharing  of  inputs  among  the  decision-makers  increased. 

Remaining  Challenges 

H6.  The  amount  of  relevant  data  available  within  the  decision  cycle  did  not 
significantly  increase. 

11.  Staffs  skills  and  knowledge  were  not  better  utilized. 

18.  The  RFI  Manager  was  not  readily  accepted  by  the  target  users  at  USPACOM. 
Usability 

Positive  Findings 

-NONE- 

Remaining  Challenges 

D 1  .Distributed  members  were  not  able  to  exchange  information  quicker. 

12.  Duplication  of  information  and  work  was  not  reduced. 

13.  Workload  was  not  reduced. 

16.  Did  not  facilitate  mission-relevant  training. 

J3.  Information  was  not  presented  in  a  fonn  that  was  usable  by  all  distributed 
members  (in  terms  of  legibility  and  feedback). 

Technical 

Positive  Findings 
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M 1 .  Server  compatibility  was  not  a  problem. 

M2.  New  applications  and  services  operated  within  existing  applications  and 
services. 

Remaining  Challenges 

A3.  Decision  support  and  knowledge  management  was  not  advanced. 

General  Findings 

Positive  Findings 

•  The  tool  generates  and  manages  RFIs  and  links  them  to  CCIRs. 

•  It  provides  search  and  automated  display  and  sorting  capabilities. 

Remaining  Challenges 

•  Many  interface  features  were  inconsistent  with  applicable  human-computer 
interface  standards. 

•  Users  cannot  determine  the  status  of  RFIs  without  opening  each  RFI. 

•  The  tool  does  not  adequately  support  multiple  user  categories  (Anonymous 
User,  Registered  User,  Manager). 

•  It  is  unavailable  to  personnel  outside  of  the  JOC. 

•  It  needs  improved  navigation  aids,  feedback,  searching,  error  prevention, 
alerting,  online  help,  screen  layout,  and  documentation. 

•  A  user-customizable  display  is  needed. 

•  Substantial  training  and  practice  are  needed  to  gain  proficiency. 

•  The  tool  may  impose  excessive  workload  on  personnel  involved  in  message 
management. 

•  The  refresh  rate  may  be  too  slow  for  operational  use. 

RFI  Manager  Summary 

While  the  concept  of  the  RFI  Manager  was  endorsed,  assessments  at  USPACOM 
indicated  that  its  current  stand-alone  status  limits  its  usefulness.  In  addition,  user 
management  issues  have  not  been  adequately  resolved.  Ill  MEF  suggested  that  dedicated 
staff  would  be  necessary  to  use  the  tool  effectively.  Therefore,  the  combined  Suitability 
rating  across  COCOM  and  Joint  Task  Force  (JTF)  levels  is  Yellow.  Assessments  at 
USPACOM  and  SSC  San  Diego  revealed  that  the  RFI  Manager  had  significant  problems 
with  usability  and  will  require  redesigns  to  become  an  effective  tool.  Its  deficiencies 
include  (1)  problems  in  determining  the  status  of  unopened  RFIs,  (2)  inconsistent 
interface  features,  and  (3)  inadequate  user  support.  Ill  MEF  reported  that  RFI  Manager 
was  used  in  operational  venues  and  suggested  capability  upgrades  that  would  enhance  its 
ease  of  use.  The  composite  rating  for  Usability  across  COCOM  and  JTF  levels  is  Yellow. 
The  Technical  component  is  rated  Green,  as  the  only  problem  encountered  during  these 
assessments  was  a  refresh  rate  that  may  be  inadequate  for  operational  use. 
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Suitability 

Usability 

Technical 

RFI  Manager 

A 

A 

© 

TeamApp 

Team  Application  (TeamApp)  is  a  special-purpose,  web-based  application  developed  at 
USPACOM  for  use  by  COCOM  and  CJTF  staff.  It  provides  point-to-point  and  multi-user  file 
transfer  and  sharing  functionality,  as  well  as  infonnation  push-pull  from  secure  and  public 
websites. 

Assessment  Events 

•  Kernel  Blitz  Experimental  KB(X)  (Jun  01).  (Appendix  R) 

TeamApp  was  evaluated  as  part  of  the  CINC  21  demonstration  during  Kernel  Blitz 
Experimental  KB(X)  in  June  2001.  Perfonnance  and  questionnaire  data  were 
collected  to  assess  the  capabilities  of  TeamApp.  A  total  of  27  participants  provided 
input  about  TeamApp,  which  included  USPACOM  staff,  COMTHIRDFLT 
personnel,  and  members  of  1  MEF. 

Suitability 

Positive  Findings 

A4.  The  consistency  of  situation  awareness  and  understanding  among  decision¬ 
makers  increased. 

14.  The  outcomes  of  distributed  work  processes  were  satisfactory  to  the  decision¬ 
makers. 

18.  TeamApp  was  accepted  by  the  target  users,  who  rated  it  favorably  in  terms  of 
utility,  power,  and  ability  to  order  information. 

Remaining  Challenges 

B3.  The  time  to  display  requested  information  was  not  decreased,  so  that 
additional  time  was  not  available  for  analyzing  decision  options. 

Usability 

Positive  Findings 

16.  TeamApp  facilitated  mission-relevant  training. 

J2.  Conveyance  of  information  on  the  display  device  platforms  was  supported. 
Remaining  Challenges 

-NONE- 

Technical 

Positive  Findings 

-NONE- 
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Remaining  Challenges 

Kl.  Controllers  had  difficulty  viewing  and  monitoring  network  operations  (due  to 
slow  download  times). 

General  Findings 

Positive  Findings 

•  It  was  useful  for  creating  and  modifying  briefings  and  for  responding  to 
CCIRs. 

•  Users  found  it  to  be  an  effective  way  to  access  the  web,  especially  for 
information  push-pull. 

•  The  interface  features  support  the  intended  functions. 

Remaining  Challenges 

•  Viewing  times  for  briefings  and  reports  are  too  slow.  (Slow  download  times 
when  bandwidth  is  limited.) 

TeamApp  Summary 

Suitability  receives  a  rating  of  Green.  It  is  well-suited  to  achieve  its  design  goals  of  file 
transfer,  creating  and  sharing  briefings,  responding  to  CCIRs,  and  information  push-pull 
from  secure  and  public  websites.  It  may  also  be  instrumental  in  reducing  manpower 
requirements.  Usability  also  receives  a  rating  of  Green.  Users  found  it  easy  to  insert 
documents  into  web  pages  and  to  communicate  with  each  other  during  the  planning 
process.  User  ratings  of  TeamApp  indicated  that  it  was  easy  to  use,  potentially  very 
powerful,  and  helpful  for  organizing  data.  Technical  is  assigned  a  rating  of  YELLOW  due 
to  the  slow  download  times  under  conditions  of  limited  bandwidth,  which  was  the  only 
negative  aspect  identified  for  TeamApp. 


Suitability 

Usability 

Technical 

TeamApp 

© 

© 

A 

Time-Based  Event  Management 

This  technology  group  consists  of  the  TMS  and  the  Master  Calendar. 

Assessment  Events 

The  same  limited  assessment  opportunity  events  (Appendix  S  and  Appendix  T)  were 
used  to  collect  data  for  technologies  in  this  group.  Data  were  gathered  from  each  user  in 
three  phases:  (1)  an  interview  in  which  users  discussed  their  likes  and  dislikes  and  how 
they  used  each  technology;  (2)  a  questionnaire;  and  (3)  a  heuristic  evaluation  based  on 
the  Style  Guide. 

Task  Management  System 

Suitability 

Positive  Findings 
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E4.  The  common  understanding  of  plan  progress  improved. 

H2.  The  completeness  of  the  information  was  sufficient  for  decision-makers' 
requirements. 

H3.  The  precision  of  the  information  was  sufficient  for  decision-makers' 
requirements. 

H4.  The  correctness  of  the  information  was  sufficient  for  the  decision-makers' 
requirements. 

B 1 .  This  technology  decreased  the  time  needed  to  collect,  identify,  and  integrate 
information. 

Remaining  Challenges 

E4.  This  technology  did  not  help  make  clear  the  dependencies  between  tasks. 

14.  This  technology  did  not  increase  satisfaction  with  work  processes. 

18.  This  technology  was  not  an  improvement  to  work  processes. 

Usability 

Positive  Findings 

D 1 .  Distributed  members  exchanged  information  quicker. 

D2.  This  technology  helped  to  decrease  the  number  of  steps  or  procedures  used  to 
exchange  information. 

El.  This  technology  decreased  the  number  of  actions  needed  to  coordinate 
between  participants. 

13.  This  technology  helped  to  decrease  workload. 

J3.  The  displayed  information  was  legible. 

Remaining  Challenges 

15.  This  technology  did  not  help  to  decrease  the  number  of  staff  needed  to 
perform  the  task. 

16.  It  was  not  easy  to  leam  how  to  use  this  technology. 

J2.  The  conveyance  of  information  on  the  display  device  platforms  was  not 
supported. 

Technical 

Positive  Findings 

Currently  in  operation. 

Remaining  Challenges 

-NONE- 

General  Findings 

Positive  Findings 
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•  This  tool  helped  to  facilitate  the  exchange  of  information. 

•  The  new  version  of  the  tool  is  more  usable  than  the  system  that  users  used  at 
time  of  assessment. 

Remaining  Challenges 

•  Users  reported  that  adequate  training  was  not  given,  and  that  this  lack  of 
training  is  responsible  for  others  using  the  system  either  incorrectly  or  not  at 
all. 

•  The  e-mail-based  system  is  hard  to  manage  short-term  processes,  because 
users  must  be  at  their  computer  to  receive  a  task  notification.  E-mail 
notification  alerting  can  be  insufficient  when  compared  to  physically  finding 
someone  and  giving  them  a  task. 

TMS  Summary 

In  the  questions  that  probed  Suitability,  many  user  responses  were  divided.  They  were 
split  on  issues  such  as  overall  satisfaction.  Based  on  questions  raised  during  the  interview 
process  regarding  TMS,  Suitability  receives  a  rating  of  YELLOW.  There  are  some 
procedural  concerns  regarding  who  is  using  the  system  and  how  they  are  using  it.  Several 
of  the  items  that  pertained  to  usability  were  rated  as  less-than-favorable  (see  Remaining 
Challenges  above).  However,  users  had  not  yet  seen  the  new  version  of  TMS,  which  was 
substantially  improved.  A  heuristic  evaluation  conducted  on  this  version  accounts  for  the 
Green  rating  for  Usability.  No  technical  problems  were  encountered,  and  the  Technical 
rating  is  Green. 


Suitability 

Usability 

Technical 

Task  Management  System 

A 

© 

© 

Master  Calendar 
Suitability 

Positive  Findings 

Al.  This  technology  helped  improve  accuracy  in  understanding  the  operational 
situation. 

B 1 .  This  technology  decreased  the  time  needed  to  collect,  identify,  and  integrate 
infonnation. 

FI.  Collaboration  improved  the  quality  of  the  plan  that  was  produced. 

HI.  The  currency  of  information  was  sufficient  for  the  decision-makers' 
requirements. 

H2.  The  completeness  of  the  information  was  sufficient  for  decision-makers' 
requirements. 

H3.  The  precision  of  the  information  was  sufficient  for  decision-makers' 
requirements. 
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H4.  The  correctness  of  the  infonnation  was  sufficient  for  the  decision-makers' 
requirements. 

14.  This  technology  increased  user  satisfaction  with  work  process. 

18.  This  technology  is  an  improvement  to  the  work  process. 

Remaining  Challenges 

E4.  The  common  understanding  of  plan  progress  was  not  improved. 

Usability 

Positive  Findings 

Dl.  Distributed  members  were  able  to  exchange  information  quicker. 

D2.  This  technology  helped  to  decrease  the  number  of  steps  or  procedures  used  to 
exchange  information. 

El.  This  technology  decreased  the  number  of  actions  needed  to  coordinate 
between  participants. 

16.  It  was  easy  to  learn  how  to  use  this  technology. 

J3.  The  displayed  information  is  legible. 

Remaining  Challenges 

13.  This  technology  did  not  help  to  decrease  workload. 

15.  This  technology  did  not  help  to  decrease  the  number  of  staff  needed  to 
perform  the  task. 

J2.  The  conveyance  of  information  on  the  display  device  platforms  was  not 
supported. 

Technical 

Positive  Findings 

This  product  is  currently  in  operation. 

Remaining  Challenges 

-NONE- 

General  Findings 

Positive  Findings 

•  Target  users  accepted  Master  Calendar. 

•  Users  found  the  interface  to  be  simple  to  use. 

•  Master  Calendar  helped  facilitate  the  exchange  of  information. 

•  Master  Calendar  provides  current  information  to  all  users. 

Remaining  Challenges 

•  More  options  are  needed  for  customizing  the  calendar  display. 
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Master  Calendar  Summary 

Master  Calendar  met  the  needs  of  its  users  and  was  very  simple  to  use.  Although  some 
improvement  could  be  made  to  offer  more  flexible  calendar  configurations,  it  firmly 
deserves  Green  ratings  across  the  board  for  Suitability,  Usability,  and  Technical. 


Suitability 

Usability 

Technical 

Master  Calendar 

© 

© 

© 

JWID  Coalition  Interoperability  Service  (COINS) 

The  Operational  Package,  JWID  COINS,  is  an  instantiation  of  Knowledge  Management,  but  it 
was  developed  as  an  application  to  specifically  support  the  Joint  Warfighter  Interoperability 
Demonstration  in  2002. 

Assessment  Events 

•  Joint  Warrior  Interoperability  Demonstration  (JWID)  2002  (Appendix  U) 

Data  were  collected  from  distributed  operational  users  in  four  coalition  countries 
(Australia,  Canada,  the  United  Kingdom,  and  the  United  States)  as  they  used  COINS 
on  the  JWID  Administrative  Portal  to  access  infonnation  from  separate  national 
databases  during  JWID  2002.  Users  also  completed  questionnaire  items  that 
addressed  various  issues  concerning  functional  utility  and  interface  usability.  In 
addition,  human  factors  professionals  perfonned  a  heuristic  evaluation  of  its  usability 
in  a  laboratory  environment. 

Suitability 

Positive  Findings 

B3.  The  time  to  display  requested  information  decreased,  so  that  additional  time 
is  available  for  analyzing  decision  options. 

Hi.  The  currency  of  information  was  sufficient  for  the  decision-makers' 
requirements. 

18.  CINC  2 1  technologies  were  accepted  by  the  target  users. 

E3.  Synchronization  of  information  management  among  key  players  (e.g.,  joint, 
coalition,  inter-agency,  and  non-governmental  organization  partners) 
increased. 

H6.  The  amount  of  relevant  data  available  within  the  decision  cycle  was 
increased. 

Remaining  Challenges 

B 1 .  The  time  needed  to  collect,  identify,  and  integrate  information  was  not 
reduced. 

12.  Duplication  of  information  and  work  was  not  reduced. 
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Usability 

Positive  Findings 

A2.  Ability  to  scale  and  tailor  visualization  displays  increased. 

13.  Workload  was  reduced. 

Remaining  Challenges 

16.  CINC  21  technologies  did  not  facilitate  mission-relevant  training. 

J3.  The  information  was  not  presented  in  a  usable  form  for  all  distributed 
members. 

Technical 

Positive  Findings 

A3.  Decision  support  and  knowledge  management  was  advanced. 

J 1 .  Infonnation  transmittal  attempts  were  successful. 

K2.  The  controllers  had  central  access  to  coalition  non-combatant  evacuation 
operation  information. 

Remaining  Challenges 

-NONE- 

General  Findings 

Positive  Findings 

•  Current  information  could  be  accessed  and  displayed  quickly  from  shared 
databases. 

•  Synchronization  of  information  management  was  achieved. 

Remaining  Challenges 

•  Currently,  entering  multiple  information  requests  requires  duplication  of 
effort. 

•  Data  cannot  be  seamlessly  transferred  between  applications. 

•  The  information  display  format  negatively  impacts  readability. 

•  The  amount  of  user  feedback  provided  by  the  system  is  inadequate. 

JWID  COINS  Summary 

Distributed  coalition  partners  saw  high  value  in  the  ability  to  access  information  from 
separate  shared  databases.  COINS  allowed  users  to  pinpoint  relevant  data  and  to 
synchronize  information  management.  The  Suitability  rating  is  Green.  The  search  process 
was  cumbersome  due  to  the  design  of  the  input  fields  and  the  need  to  duplicate  efforts  to 
conduct  multiple  single  searches.  The  results  display  was  not  easily  readable  due  to  poor 
design  of  the  format.  Users  were  unable  to  seamlessly  transfer  data  to  other  applications, 
and  had  to  spend  much  time  cutting  and  pasting.  Due  to  these  issues,  Usability  is  rated  as 
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YELLOW.  The  availability  of  COINS  and  accessibility  of  data  was  high.  The  Technical 
rating  is  Green. 


Suitability 

Usability 

Technical 

JWID  COINS 

© 

A 

© 

Displays 

Group  Displays 

The  Group  Displays  contain  assessments  of  the  USSTRATCOM  Knowledge  Wall  and  the 
USPACOM  Video  Wall.  The  Video  Wall  data,  however,  are  very  limited.  More  importantly,  the 
Video  Wall  was  an  early  implementation  of  a  large-screen  display.  It  has  very  limited  utility  and 
is  not  a  candidate  for  transition. 

USPACOM  Video  Wall  (Standard  Resolution  Large  Screen  Display) 

Assessment  Events 

•  Build  1  Technologies  Assessment  Report  (October  2000)  (Appendix  V) 

•  Limited  HCI  analysis 
Suitability 

Positive  Findings 

-NONE- 

Remaining  Challenges 

Al.The  accuracy  of  situation  awareness  and  understanding  among  decision¬ 
makers  was  not  increased. 

H2.  The  completeness  of  the  information  was  not  sufficient. 

Usability 

Positive  Findings 

D2.  The  number  of  steps  or  procedures  used  to  exchange  information  was 
reduced. 

J2.  Conveyance  of  information  on  the  display  device  platforms  was  supported. 

J3.  Information  was  presented  in  a  form  that  was  usable  by  all  distributed 
members. 

Remaining  Challenges 

A2.  Ability  to  scale  and  tailor  visualization  displays  was  not  increased. 

D2.  The  number  of  steps  or  procedures  used  to  exchange  information  was  not 
reduced. 

13.  Workload  was  not  reduced. 
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17.  Watchstanders  were  not  able  to  relay  information  faster  when  managing 
routine  theater  activity. 

Technical 

Positive  Findings 

-NONE- 

Remaining  Challenges 

K5.  The  system  did  not  provide  a  secure  means  for  exchanging  information 
among  distributed  users. 

General  Findings 

Positive  Findings 

•  The  Video  Wall  provided  a  focal  point  for  group  viewing  and  collaboration. 

•  Target  users  accepted  this  technology. 

Remaining  Challenges 

•  The  display  needs  to  be  designed  to  scale  properly. 

•  It  increased  the  potential  for  information  overload. 

•  Text  size  and  content  layout  needs  to  be  designed  to  support  group  viewing 
and  attention  management. 

•  Physical  room  layout  and  ambient  lighting  negatively  impact  visibility  of  the 
Video  Wall. 


USPACOM  Video  Wall  Summary 

The  USPACOM  Video  Wall  represents  early  large-screen  display  technology.  It  only 
provides  three  independent  viewing  screens.  The  input  to  these  screens  can  be  either 
digital  or  video.  In  addition,  although  it  is  currently  used  to  provide  support  during 
command  briefings,  the  Video  Wall  is  not  planned  for  continued  use  by  USPACOM. 
Thus,  any  rating  would  be  inappropriate. 


Suitability 

Usability 

Technical 

PACOM  Video  Wall 

<t) 

USSTRATCOM Knowledge  Wall  (High-Resolution,  Large-Screen  Display) 

Assessment  Events 

•  Limited  Assessment  Opportunity  to  determine  the  legibility  of  various  text  sizes 
at  various  fixed  viewing  distances  (Appendix  X) 

•  Interviews  of  expert  Knowledge  Wall  operators  (producers) 
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•  USSTRATCOM  Consequence  Management/Response  Demonstration  (October 
2002)  (Appendix  I) 

Suitability 

Positive  Findings 

Al.  The  accuracy  of  situation  awareness  and  understanding  among  decision¬ 
makers  was  increased. 

B 1 .  The  time  needed  to  collect,  identify,  and  integrate  information  was  reduced. 

B3.  The  time  to  display  requested  information  decreased,  so  that  additional  time 
is  available  for  analyzing  decision  options. 

H2.  The  completeness  of  the  information  was  sufficient  for  decision-makers' 
requirements. 

H6.  The  amount  of  relevant  data  available  within  the  decision  cycle  increased. 
18.  CINC  2 1  technologies  were  accepted  by  the  target  users. 

Remaining  Challenges 

-NONE- 

Usability 

Positive  Findings 

A2.  Ability  to  scale  and  tailor  visualization  displays  increased. 

D2.  The  number  of  steps  or  procedures  used  to  exchange  information  was 
reduced. 

13.  Workload  was  reduced. 

15.  The  number  of  staff  needed  to  perfonn  the  task  was  reduced. 

17.  Watchstanders  were  able  to  relay  information  faster  when  managing  routine 
theater  activity. 

J2.  Conveyance  of  information  on  the  display  device  platforms  was  supported. 

J3.  Information  was  presented  in  a  fonn  that  was  usable  by  all  distributed 
members. 

LI.  The  time  was  reduced  in  establishing  a  collaborative  framework. 

Remaining  Challenges 

-NONE-- 

Technical 

Positive  Findings 

A3.  The  decision  support  and  knowledge  management  was  advanced. 

J 1 .  Infonnation  transmittal  attempts  were  successful. 
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Remaining  Challenges 

H2.  The  completeness  of  the  information  was  not  sufficient 

K5.  The  system  did  not  provide  a  secure  means  for  exchanging  information 
among  distributed  users. 

General  Findings 

Positive  Findings 

•  This  display  gave  users  the  “big  picture”  view,  increasing  situation  awareness. 

•  It  provided  a  focal  point  for  group  viewing  and  collaboration. 

•  Excellent  visibility  of  text  and  graphics  was  noted  from  most  viewing 
positions. 

•  Target  users  accepted  the  technology. 

Remaining  Challenges 

•  The  display  needs  to  be  designed  to  scale  properly  to  Knowledge  Wall. 

•  There  is  increased  potential  for  information  overload. 

•  Text  size  and  content  layout  needs  to  be  designed  to  support  group  viewing 
and  attention  management. 

•  Physical  room  layout  and  ambient  lighting  could  negatively  impact 
Knowledge  Wall  visibility. 

USSTRATCOM  Knowledge  Wall  Summary 

The  Knowledge  Wall  has  been  in  use  at  USSTRATCOM  for  approximately  1  year,  and 
they  have  had  time  to  discover  many  pros  and  cons  regarding  its  use  in  an  operational 
setting.  Having  such  a  large  screen,  there  are  many  areas  of  concern  regarding  interface 
layout,  room  layout,  information  overload,  and  attention  management.  However,  at  the 
USSTRATCOM  demonstration  in  October  2002,  participants  commented  that  the 
Knowledge  Wall  improved  understanding  of  the  operational  situation  and  that  it  was  easy 
to  follow  the  briefings.  These  high  user  endorsements,  coupled  with  a  lack  of  technical 
problems,  accounts  for  Green  ratings  in  Suitability,  Usability,  and  Technical. 


USSTRATCOM  Knowledge 
Wall 

Suitability 

© 

Usability 

© 

Technical 

© 

Workstation  Displays 

The  Workstation  Displays  technology  category  consists  of  multi-screen  displays.  Workstation 
Displays  can  be  implemented  either  as  separate  monitors  placed  side-by-side  or  as  a  single  multi¬ 
screen  display,  such  as  the  Panoram  PV290. 
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Multiple  Side-by-Side  Flat  Panel  Monitors 
Assessment  Events 

•  Human-system  interface  analysis 

•  Field  observations  and  user  feedback 
Suitability 

Positive  Findings 

Al.  The  accuracy  of  situation  awareness  and  understanding  among  decision¬ 
makers  was  increased. 

B 1 .  The  time  needed  to  collect,  identify,  and  integrate  information  was  reduced. 

B3.  The  time  to  display  requested  information  decreased,  so  that  additional  time  is 
available  for  analyzing  decision  options. 

H2.  The  completeness  of  the  information  was  sufficient  for  decision-makers' 
requirements. 

H6.  The  amount  of  relevant  data  available  within  the  decision  cycle  increased. 

18.  CINC  21  technologies  were  accepted  by  the  target  users. 

Remaining  Challenges 

-NONE- 

Usability 

Positive  Findings 

A2.  Ability  to  scale  and  tailor  visualization  displays  increased. 

13.  Workload  was  reduced. 

15.  The  number  of  staff  needed  to  perform  the  task  was  reduced. 

J2.  Conveyance  of  information  on  the  display  device  platforms  was  supported. 

J3.  The  information  was  presented  in  a  form  that  was  usable  by  all  distributed 
members. 

Remaining  Challenges 

-NONE- 

Technical 

Positive  Findings 

A3.  The  decision  support  and  knowledge  management  was  advanced. 

Jl.  Information  transmittal  attempts  were  successful. 

Remaining  Challenges 

-NONE- 
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General  Findings 

Positive  Findings 

•  The  display  configuration  possesses  substantial  technical  capability  to  support 
concurrent  processes. 

•  It  affords  increased  situation  awareness  and  integration  of  diverse  information. 

•  The  greater  screen  space  improves  the  user’s  ability  to  detect  critical  events 
and  changes. 

•  It  enables  easier  access  to  diverse  information  sources  and  software 
applications. 

•  It  supports  faster  responses  to  multiple  collaboration  (chat)  sessions. 

Remaining  Challenges 

•  It  requires  careful  layout  of  workspaces  and  applications  onto  screens. 

•  Video  perfonnance  (speed)  may  be  degraded  when  highly  dynamic 
applications  are  used. 

Multiple  Side-by-Side,  Flat-Panel  Monitors 

With  ergonomically  acceptable  setup  and  adequate  technical  support  (e.g.,  appropriate 
video  cards),  the  multiple  side-by-side  flat-panel  monitors  provide  many  benefits  to  the 
user  at  a  relatively  low  cost.  This  type  of  workstation  display  earns  Green  ratings  for 
Suitability,  Usability,  and  Technical. 


Multiple  Side-by-Side  Flat 
Panel  Monitors 

Suitability 

© 

Usability 

© 

Technical 

© 

PV290  Multi-screen  Desktop  Display 

The  Panoram  PV290  DSK  desktop  tri-panel  LCD  display  is  an  industrial-grade  display  with 
12  source  inputs,  external  USB  input  switching,  and  built-in,  multi-channel  audio.  The 
overall  image  size  across  its  three  equal-sized  screens  (18.1  inches  diagonal)  is  43.5  inches  x 
1 1.5  inches,  making  it  approximately  equivalent  to  three  19-inch  monitors.  The  resolution  of 
each  screen  is  1280  x  1024  (overall  resolution  =  3840  x  1024). 

The  PV290  is  an  example  of  the  type  of  multi-screen  monitor  that  may  be  suitable  for 
military  information  technology  and  command  and  control  tasks.  However,  the  information 
architecture  of  CINC  21  (specifically  DFC2)  did  not  utilize  the  full  capabilities  of  the  PV290 
(e.g.,  simultaneous  handling  of  inputs  from  up  to  six  computers  and  three  monitors  that 
provided  six  VGA  screens  and  three  NTSC). 
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Assessment  Event 

An  ergonomic  evaluation  of  the  PV290  was  conducted  using  the  ergonomic  guidelines 
and  standards  contained  in  “Ergonomic  Guidelines  for  Multiple  Monitor  Workstations.”9 
Because  no  actual  PV290  display  was  available  for  this  evaluation,  this  assessment  was 
based  on  the  descriptions,  specifications,  and  graphical  images  of  the  PV290  found  on  its 
web  page  (http://www.panoramtech.com/products/pv290.html).  Although  there  are 
inherent  limitations  in  not  directly  examining  an  actual  PV290,  it  is  still  possible  to  assess 
its  overall  ergonomic  characteristics  relative  to  CINC  21  requirements.  This  ergonomic 
assessment  was  supplemented  with  informal  comments  provided  by  PV290  users  at 
USPACOM  and  USSTRATCOM. 

Overall,  the  PV290  affords  the  same  benefits  and  challenges  as  noted  previously  for  the 
multiple,  side-by-side  flat  panel  monitors.  Since  the  PV290  provided  three  screens  on 
which  users  could  display  a  variety  of  task-relevant  information,  it  makes  it  easier  for 
users  to  detect  changes  in  critical  events,  access  and  integrate  diverse  information 
sources,  and  maintain  situation  awareness.  On  the  other  hand,  some  effort  is  needed  with 
any  multi-monitor  workstation  to  layout  the  information  on  the  screens  properly  and  to 
train  users  to  manipulate  the  screen  configurations.  The  following  are  additional  issues, 
specific  to  the  PV290. 

General  Findings 

Positive  Findings 

•  The  specifications  of  the  PV290  indicate  that  it  satisfies  the  single-user 
ergonomic  guidelines  for  monitor  display  properties,  such  as  screen 
luminance,  contrast,  color,  and  luminance  uniformity. 

•  The  three  screens  are  separated  by  a  0.625-inch  wide  bezel,  enabling  it  to 
present  a  more  seamless  image  than  separate,  conventional  monitors  placed 
side-by-side. 

•  The  total  lateral  viewing  angle  of  the  three  screens  is  less  than  190  degrees,  in 
accordance  with  ergonomic  guidelines. 

•  The  three  screens  are  approximately  equidistant  from  the  viewer  when  seated 
at  a  recommended  distance  from  the  center  screen  (25-30  inches). 

•  Because  the  three  screens  are  approximately  equidistant  from  the  viewer, 
moving  from  one  screen  to  another  can  be  done  with  little  effect  on  visual 
accommodation  and  convergence,  consistent  with  ergonomic  guidelines. 

Remaining  Challenges 

•  The  non-adjustable  viewing  angles  make  it  difficult  for  more  than  one  person 
to  view  the  screens  simultaneously. 

•  The  dimensions  of  the  PV290  (height:  19  inches,  width:  43  inches,  depth:  22 
inches)  combine  with  its  canted  peripheral  screens  to  require  a  large  desktop 


9  Pacific  Science  &  Engineering  Group.  2000.  Ergonomic  Guidelines  for  Multiple  Monitor  Workstations. 
San  Diego,  CA. 
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footprint.  This  footprint  may  limit  the  available  working  surface  for  papers, 
etc. 

•  Sub-optimal  positioning  of  the  PV290  due  to  its  large  footprint  could  result  in 
glare,  diminished  screen  contrast,  and  specular  reflectance. 

•  The  PV290  is  expensive  compared  to  the  cost  of  three  conventional  monitors. 

•  The  users’  tasks  and  work  processes  do  not  currently  require  the  full 
capability  of  the  PV290.  Job  redesign  and  training  analysis  are  needed  to 
determine  how  its  capabilities  could  be  used  in  an  operational  work  setting.  If, 
however,  expanded  work  processes  are  determined  to  be  unnecessary,  then  the 
PV290  capability  would  be  assessed  as  excessive. 

•  Considerable  specialized  maintenance  by  the  manufacturer  is  needed,  and 
quick  access  to  unique  replacement  parts  is  necessary  in  a  military  setting. 
Transfer  of  some  of  these  repair  and  maintenance  functions  to  onsite  staff 
would  be  highly  beneficial  in  terms  of  time  and  cost. 

PV290  Multi-Screen  Desktop  Display  Summary 

The  PV290  satisfies  most  ergonomic  guidelines  for  desktop  displays  viewed  by  a  single 
user  who  is  seated  in  the  recommended  viewing  position.  It  is  less  satisfactory  for 
multiple  simultaneous  viewers,  not  all  of  who  will  be  able  to  assume  acceptable  viewing 
positions  in  relation  to  the  three  LCD  screens.  Given  the  high  cost  of  a  PV290  and  the 
fact  that  present  CINC  21  technologies  do  not  take  full  advantage  of  the  PV290’s 
capabilities  (e.g.,  handling  simultaneous  inputs  from  multiple  computers  and  monitors), 
its  use  is  not  currently  recommended  for  CINC  2 1  activities. 


PV290  Multi-Screen 

Suitability 

Usability 

Technical 

Desktop  Display 

© 

© 

A 

Collaboration 

This  section  contains  the  assessments  of  the  primary  collaborative  services  that  have  been 
provided  by  the  CINC  21  core  capabilities  or  have  been  incorporated  as  components  of  CINC  21 
technologies.  Most  of  these  are  provided  by  the  Defense  Collaboration  Tool  Set  (DCTS)  or  its 
components,  NetMeeting  and  the  Distributed  Collaborative  Planning  and  Conferencing  (DCP) 
feature  of  the  JWID  Administration  Portal.  Due  to  the  non-independent  nature  of  these  tools, 
data  from  these  technology  sets  are  combined  in  this  section.  In  addition,  data  from  IWS,  which 
is  used  extensively  at  USPACOM,  are  included.  The  IWS  data,  however,  were  limited. 

Assessment  Events 

•  Build  1  Technologies  Assessment  Report  (October  2000)  (Appendix  V) 

•  Kernel  Blitz  Experimental  (KB(X)  (Oct  01)  (Appendix  R) 

•  CINC  21  Training  (March  202)  (Appendix  W) 

•  Joint  Warrior  Interoperability  Demonstration  (May  2002)  (Appendix  U) 
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•  USSTRATCOM  Consequence  Management/Response  Demonstration  (October 

2002)  (Appendix  j) 

•  Special  assessment  by  J2 1  USPACOM  (February  2003) 

Data  were  collected  from  operational  users  during  multiple  events.  Demonstration/ 
training  and  hand-on  usage  of  the  technology  were  components  of  the  sessions.  User 
feedback  and  observation  by  assessors  were  the  primary  data  collection  methods 
used. 

Suitability 

Positive  Findings 

Al.The  accuracy  of  situation  awareness  and  understanding  among  decision¬ 
makers  increased. 

A4.The  consistency  of  situation  awareness  and  understanding  among  decision¬ 
makers  increased. 

B3.The  time  to  display  requested  information  decreased,  so  that  additional  time 
was  available  for  analyzing  decision  options. 

Cl. Planners'  satisfaction  with  the  decision  increased. 

E2.  Synchronization  of  decision  tasks  among  key  players  (e.g.  joint,  coalition, 
inter-agency,  and  non-governmental  organization  partners)  improved. 

E3.  Synchronization  of  information  management  among  key  players  (e.g.  joint, 
coalition,  inter-agency,  and  non-governmental  organization  partners) 
increased. 

E4.  Common  understanding  of  plan  progress  improved. 

E6.  Team  actions  were  better  synchronized  by  collaboration  (IWS  only). 

F2.  The  sharing  of  inputs  among  the  decision-makers  increased. 

Gl.The  time  to  develop  of  mission  and  intent  was  reduced. 

12.  Duplication  of  information  and  work  was  reduced  (IWS  only). 

13.  Workload  was  reduced  (IWS  only). 

14.  Outcomes  of  distributed  work  processes  were  satisfactory  to  the  decision¬ 
makers. 

18.  CINC  21  technologies  were  accepted  by  the  target  users. 

Remaining  Challenges 

B2.Time  availability  to  analyze  the  information  and  determine  the  best  CO  A  was 
not  reduced. 

15.  The  number  of  staff  needed  to  perform  the  task  was  not  reduced  (IWS  only). 

L2.  The  response  time  to  decision-makers’  questions  was  not  decreased. 
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Usability 

Positive  Findings 

Dl. Distributed  members  exchanged  information  quicker. 

D2.The  number  of  steps  or  procedures  used  to  exchange  infonnation  was  reduced. 

El.  The  number  of  actions  needed  to  coordinate  between  participants  was 
reduced. 

J3.  The  information  was  presented  in  a  form  that  is  usable  by  all  distributed 
members. 

Remaining  Challenges 

A2.  Ability  to  scale  and  tailor  visualization  displays  was  not  increased. 

16.  CINC  21  technologies  did  not  facilitate  mission-relevant  training. 

Technical 

Positive  Findings 

K5.  The  system  provided  a  secure  means  for  exchanging  information  among 
distributed  users. 

Ml .  Server  compatibility  was  not  a  problem  (IWS  only). 

Remaining  Challenges 

A3.  Decision  support  and  knowledge  management  was  not  advanced. 

J 1 .  Information  transmittal  attempts  were  not  successful 
J4.  Conveyance  of  information  was  not  facilitated. 

K2.  The  controllers  often  did  not  have  central  access  to  coalition  non-combatant 
evacuation  operation  infonnation. 

M 1 .  Server  compatibility  was  a  problem. 

M2.  New  applications  and  services  did  not  operate  within  existing  applications 
and  services. 

General  Findings 

Positive  Findings 

•  Shared  applications  supported  interaction  among  users. 

•  Collaborative  sessions  between  distributed  users  led  to  increased  situation 
awareness. 

•  Collaborative  tools  are  considered  useful  when  consistently  available. 

•  Collaborative  sessions  facilitate  efficient  information  exchange. 

•  Secure  collaboration  can  be  provided  for  up  to  120  participants  (IWS  only). 
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Remaining  Challenges 

•  Stable  connectivity  must  be  provided. 

•  Improved  user  feedback  is  needed  in  areas  of  login,  active/available  sessions, 
active  members,  and  status  of  audio/video. 

•  Training  and  operating  procedures/business  rules  are  needed  for  efficient  use 
of  collaboration  tools. 

Collaboration  Summary 

Distributed  users  found  high  value  in  sharing  documents,  briefs,  and  desktops.  They 
considered  that  collaboration  tools  provide  an  efficient  mechanism  for  the  exchange  of 
information.  Users  judged  that  collaborative  sessions  led  to  increased  situation 
awareness.  Suitability  summary  is  rated  as  Green. 

However,  chat  often  became  the  de  facto  collaboration  tool  due  to  bandwidth  restrictions. 
Audio  was  preferred  when  available;  video  was  often  unstable  and  considered  by  many 
users  to  be  redundant  and  unnecessary.  The  status  of  audio-video  and  of  session 
membership  was  frequently  difficult  to  discern  with  the  DCTS  technologies.  Feedback  to 
the  user  regarding  success  of  data  transmittals  was  not  always  adequate.  In  addition, 
several  areas  of  the  IWS  computer  interface  were  identified  as  requiring  modification  to 
improve  ease  of  use.  Other  impediments  to  use  are  lack  of  training,  lack  of  business  rules, 
and  lack  of  interoperability  between  collaboration  systems.  Usability  is  rated  as  YELLOW. 

The  inability  of  DCTS  technologies  to  provide  stable  connectivity  is  an  obstacle  toward 
reaching  the  goal  of  synchronizing  decision-making  and  information  management  among 
distributed  users.  In  addition  to  the  technical  aspects,  personnel’s  confidence  and 
willingness  to  use  these  collaboration  tools  is  negatively  impacted.  However,  the  ability 
to  provide  stable  connectivity  was  demonstrated  with  IWS  in  actual  operational  use. 
Therefore,  the  Technical  summary  indication  for  the  CINC  21  collaborative  services 
technology  category  is  Yellow. 


Suitability 

Usability 

Technical 

Collaboration 

© 

A 

A 

Network  Operations  Tools  and  Services 

The  network  operations  aggregate  several  commercial  applications  through  the  CINC  21 
knowledge  core.  The  applications  are  ingested  in  the  knowledge  core  with  their  XML  tags.  This 
infonnation  is  then  aggregated  and  displayed  as  tables  (in  near  real-time)  via  business  rules.  The 
presentation  is  a  matrix  with  drill-down  capability  that  presents  the  state  of  operational  servers 
and  networks.  Network  operations  were  supported  by  monitoring  and  infonnation  assurance 
tools.  Assessment  of  the  Network  Operations  suite  is  provided  below.  Separate  assessments  are 
also  provided  for  the  TCCC  Status  Grid,  with  a  focus  on  the  output  of  the  display,  and  for  BMC 
Patrol,  as  this  tool  has  been  installed  and  fielded  independently. 
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Theater  C4ISR  Status  Grid 

This  grid  is  a  graphic  display,  broken  down  by  theater  organization  and  type  of  resource.  It 
displays  a  rollup  of  the  Remedy  trouble  tickets  assigned  to  each  location  and  resource.  A 
drill-down  capability  allows  the  operator  to  see  the  actual  ticket. 

Assessment  Events 

Assessment  was  addressed  during  the  Build  1  Technologies  Assessment  (Appendix  V), 
the  JJWID  2002  (Appendix  U),  and  the  assessment  of  the  CINC  21  Technology  DFC2 
Interface  (Appendix  E  and  Appendix  F).  A  depiction  of  the  TCCC  Status  Grid  was 
available  during  JWID  and  USPACOM  assessments,  but  viewing  the  operational  status 
of  networks  was  not  possible  as  there  was  no  active  link  to  the  display.  Changes  to  the 
display  were  accomplished  manually.  However,  comments  by  representative  users  about 
this  display  were  collected.  Assessment  was  also  provided  by  users  in  the  TCCC 
(February  2003).  These  data  are  integrated  into  the  following  lists. 

Suitability 

Positive  Findings 

HI  Information  was  current. 

H2.  Information  was  relevant. 

H3.  Information  was  precise. 

H4.  Information  was  accurate. 

Remaining  Challenges 

B 1 .  Did  not  decrease  time  needed  to  collect,  identify,  and  integrate  information. 
1 1 .  Did  not  fit  well  with  current  work  processes. 

14.  Did  not  increase  satisfaction  with  work  processes. 

Usability 

Positive  Findings 

D 1 .  Facilitated  information  exchange. 

J2.  It  was  easy  to  change  layout. 

J2.  It  was  easy  to  change  content. 

J3.  Information  was  legible. 

Remaining  Challenges 

D2.  Did  not  decrease  number  of  steps  or  procedures  used  to  exchange 
infonnation. 

13.  Did  not  decrease  workload. 

16.  Time  to  learn  the  technology  would  be  excessive. 
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Technical 

Positive  Findings 

J1 .  Information  transmittal  attempts  were  successful. 

Remaining  Challenges 

K1 .  Controllers  were  not  able  to  increase  their  ability  to  view  and  monitor 
network  operations. 

Theater  C4ISR  Status  Grid  Summary 

Provision  of  information  by  the  grid  was  highly  rated.  However,  users  reported  no 
increase  in  saving  time  [SRBijto  share  the  information;  TCCC  user  acceptance  was  mixed. 
Users  in  the  JOC  reported  that  the  display  would  have  limited  value  for  them.  Suitability 
rates  a  Yellow.  Workload  was  not  decreased  for  users  in  the  TCCC.  Users  in  the  JOC 
estimated  a  steep  learning  curve  for  the  display  to  be  useful.  Usability  is  rated  Yellow. 
TCCC  users  reported  that  information  was  successfully  transmitted  via  the  grid,  but  its 
use  did  not  increase  their  ability  to  view  and  monitor  network  operations,  nor  did  it 
increase  their  confidence  in  system  security.  Technical,  therefore,  rates  a  Yellow.  The 
composite  scores  for  TCCC  Status  Grid  are  shown  below. 


Suitability 

Usability 

Technical 

Theater  C4ISR  Status  Grid 

A 

A 

A 

Network  Operations  (NetOps)  Status  Monitoring 

The  NetOps  suite  provides  real-time  status  of  network  resources  and  services  across  the 
theater  using  a  set  of  portal  based  visualizations  viewable  from  within  a  web  browser.  These 
visualizations  are  designed  to  help  the  TCCC  Watch  Commander  rapidly  determine 
COCOM-level  mission  impact  of  network  and  resource  outages  from  across  the  entire 
theater. 

The  visualizations  are  divided  into  two  distinct  categories: 

•  Infonnation  Assurance 

o  Derives  their  displays  from  data  generated  by  the  AAIDE  system, 
o  Leverages  AIDE  System’s  ability  to  collect  and  nonnalize  intrusion  events, 
o  Includes  the  EICS,  EPC,  and  Link  Analysis  tools. 

•  Network  Monitoring 

o  Derives  their  displays  from  data  generated  by  the  Remedy  Trouble  Ticket  system 
using  a  specifically  formatted  trouble  ticket, 
o  Uses  Remedy  to  drive  displays  and  to  integrate  with  other  system  tools, 
o  Includes  the  TCCC  Status  Grid  and  Automated  Capability  Brief  tools. 

The  goals  for  providing  these  products  are  as  follows: 

•  Improve  the  TCCC  Watch  Officer’s  ability  to  provide  up-to-date  mission  impact 
assessment  to  the  Command  Center. 


85 


Joint  Military  Utility  Assessment 


Provide  the  TCCC  Watch  Officer  the  ability  to  take  twice-daily  snapshots  of  the  entire 
theater’s  network  resources  status  in  the  fonn  of  an  editable,  navigable  HTML  brief. 
These  briefs  are  shown  to  the  senior  decision-makers  and  archived  for  future  reference. 

Provide  the  Command  Center  with  high-level,  near-real-time  status  of  theater  network 
resources  in  the  form  of  automated  network  operations  CCIR  status. 

Fielding 

The  NetOps  suite  is  implemented  as  a  web  application  and  is  currently  used  by  the 
USPACOM  TCCC  in  its  stand-alone  form. 

The  NetOps  suite  of  tools  has  come  from  an  evolutionary  process  of  step-wise 
refinement.  There  exists  no  requirements  documentation  describing  the  desired  ultimate 
functionality.  This  process  allows  for  rapid  application  development  but  also  injects  a 
great  amount  of  change  into  the  software  engineering  process.  There  exists  a  tradeoff 
between  flexibility,  environmental  requirements,  and  the  time  needed  to  develop  a 
functional  demonstration  as  each  step  of  this  evolutionary  process  unfolds. 

Positive  Findings 

•  The  operators  in  the  USPACOM  TCCC  use  the  network  monitoring  tools, 
o  TCCC  Status  Grid 

o  Impact  Statement  editor/display 
o  Automated  Brief  (on  hold) 

•  When  AIDE  data  become  available,  then  the  AIDE  visualization  tools  will  be 
live. 

•  Two  major  architectural  pieces  have  been  built.  The  NetOps  visualization 
engine  (Figure  5),  and  the  OneWay  Proxy  (Figure  6).  The  proxy  uses  a  plug¬ 
in  architecture.  Plug-ins  for  AIDE  and  BMC  Patrol  have  been  implemented. 
These  pieces  can  be  quickly  adapted  to  support  other  forms  of  data  as  an  input 
source. 

•  The  portal  approach  allowed  several  different  presentations  to  be  quickly 
implemented. 

o  Flexibility  of  display  and  screen  space  usage 

o  XML  Application  Program  Interface  (API)  for  external  access  to  NetOps 
generated  data 
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Remedy  Tccc 


Figure  6.  One-Way  Proxy. 
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Remaining  Challenges 

•  Continuous  access  to  an  AIDE  data  stream  has  not  been  achieved  either  locally  in 
the  AITS-JPO  or  at  USPACOM.  This  renders  the  information  assurance 
visualizations  inactive. 

o  Events  Involving  Critical  Servers 
o  EPC 

o  Link  Analysis 

•  The  Continuously  Computed  Impact  Point  (CCIP)  at  USPACOM  was  flooded  by 
a  broken  water  pipe.  The  flood  destroyed  several  computers,  including  the 
computer  that  supported  NetOps.  NetOps  has  since  been  deployed  in  the  IEC  at 
AITS-JPO  but  was  not  designed  to  adequately  handle  the  large  latency  times 
encountered  as  packets  move  between  AITS-JPO  and  USPACOM.  The 
processing  logic  needs  to  be  decoupled  from  the  data  mining  logic. 

Recommendations  for  Future 

•  Standardize  the  trouble  ticket  used  by  the  entire  Pacific  theater.  This  change 
would  allow  all  the  organizations  to  quickly  communicate  network  status  to  the 
TCCC. 

•  Spend  more  time  with  the  operators  in  the  TCCC  to  gain  a  better  understanding  of 
their  business  processes. 

•  Fully  define  the  appropriate  integration  points  between  DFC2  and  NetOps. 

•  Provide  a  business  logic  module  that  would  allow  an  operator  to  define  simple 
business  logic  status  rules. 

•  Provide  a  module  that  would  allow  the  operator  to  author  automated  briefs 

Net  Ops  Status  Monitoring  Summary 

The  EICS  is  an  advanced  visualization  that  provides  the  ability  to  track  statistics  about 
attacks  on  critical  servers.  EICS  is  a  valuable  tool  for  network  operations  and  is  rated 
Green  for  Suitability.  It  is  rated  Yellow  for  Usability  and  Technical  because  the 
visualization  display  needs  to  be  reformatted  to  facilitate  better  comprehension  by  the 
users,  and  it  is  currently  inactive  at  USPACOM. 

The  EPC  is  a  stacked  bar  chart  visualization  showing  Information  Assurance  (IA)  status 
of  AIDE  system  over  the  last  30  days.  EPC  is  rated  Green  for  Suitability  and  Usability 
because  it  provides  a  quick  snapshot  visualization  of  IA  activity,  as  well  as  drill-down 
capability  to  view  details  of  the  individual  events.  It  is  Yellow  for  Technical  because  it  is 
currently  inactive  at  USPACOM. 

The  Link  Analysis  is  an  advanced  visualization  showing  IA  status  of  attackers/attackees. 
The  ability  to  draw  information  from  IA  sensors  and  track  developments  is  very 
important.  Thus,  Suitability  for  Link  Analysis  is  rated  Green.  It  is  rated  Yellow  for 
Usability  and  Technical  because  the  visualization  tends  to  become  crowded  with  data  and 
requires  additional  filtering  capability. 
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The  composite  ratings  for  Net  Ops  Status  Monitoring  are  shown  below.  The  various 
capabilities  are  viewed  as  valuable  assets  for  the  TCCC  and  for  keeping  track  of  the 
Theater  network  status.  Thus,  Suitability  is  rated  Green.  Given  the  problems  associated 
with  implementation  and  integration  of  the  tools,  Usability  and  Technical  are  rated 
Yellow. 


Suitability 

Usability 

Technical 

Net  Ops  Status  Monitoring 

© 

A 

A 

BMC  Patrol 

Providing  enterprise  network  management  capabilities  that  can  monitor  the  status  of  key 
servers  and  the  health  of  the  network,  as  well  as  the  ability  to  interface  with  other  network 
management  tools  such  as  Tivoli®,  Hewlett-Packard  OpenView®,  and  Remedy®,  is  an 
important  network  infrastructure  activity.  PATROL®  was  identified  as  a  Gold  Nugget  by  the 
JWID2000. 

Current  network  operations  utilize  highly  skilled  technicians  actively  monitoring  computer 
and  communication  systems  for  failures.  Usually,  hardware,  software,  or  network  failures  are 
discovered  by  the  customers,  reported  to  a  help  desk  facility,  which  records  the  problem 
report  and  forwards  it  either  manually  or  automatically  for  resolution.  PATROL®  is  an 
automated  discovery  process  that  understands  not  only  when  major  failures  occur,  but  also 
interprets  the  accumulation  of  minor  events  that  lead  to  customer-service-level  degradations. 
These  same  processes  can  automatically  recover  from  minor  failures  without  demanding  the 
time  and  skills  of  system  administrators.  This  ability  will  fonn  a  service-level  management 
model  that  is  focused  on  the  customer’s  requirements.  PATROL®  hardware,  software,  and 
processes  define  customer  centered  service  level  management. 

PATROL®  performs  the  following  functions: 

•  Collects  and  analyzes  real-time  events  and  data. 

•  Diagnoses  and  resolves  problems  before  end-users  are  affected. 

•  Optimizes  system  performance  and  prevents  outages. 

•  Reports  deviations  from  expected  service  levels. 

•  Predicts  how  changes  in  the  environment  will  affect  future  service  levels. 

Fielding 

The  CINC  21  ACTD  assisted  in  installation  and  fielding  of  PATROL®  at  the  AITS-JPO  and 
at  USPACOM.  The  following  components  were  installed,  demonstrated,  and  configured: 

Enterprise  Management 

•  PATROL®  Enterprise  Manager  (PATROL®  EM) 

•  PATROL  Enterprise  Manager  Agent  Connection 
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•  PATROL®  Integration  for  Hewlett-Packard  Open  View'  Network  Node  Manager 

(AITS-JPO  only) 

•  PATROL®  Integration  for  Tivolli®  (AITS-JPO  only) 

•  PATROL®  Integration  for  Remedy®  Action  Request  System  (ARS)  (AITS-JPO  only) 

•  PATROL®  Operations  Manager  (AITS-JPO  only) 

•  PATROL®  Service  Reporting  (AITS-JPO  only) 

•  PATROL®  Explorer  (Console) 

Application  Service  Management 

Operating  System  Layer 

•  PATROL®  for  Microsoft  Windows®  2000  Server 

•  PATROL®  for  Unix 

Applications/Services  Layer 

•  PATROL®  for  Microsoft®  Exchange  Server  (AITS-JPO  only) 

•  PATROL®  for  Microsoft®  Structured  Query  Language  (SQL)  Server  (AITS-JPO 

only) 

•  PATROL®  for  Internet  Services 

•  PATROL®  for  Oracle®  (AITS-JPO  only) 

•  CINC  2 1  Developed  Knowledge  Module  for  CuSeeMe™  (AITS-JPO  only) 

Console 

•  PATROL®  Console 

Positive  Findings 

•  PATROL®  installation  at  the  AITS-JPO  showcased  total  enterprise  monitoring 
capability  between  Hewlett-Parckard  OpenView®  (HPOV),  Remedy®,  and 
Tivolli®. 

•  During  JWID  2002  PATROL®  integration  with  HPOV  and  Remedy®  was  used  to 
generate  network  status  matrices  automatically. 

•  One-way  feeds  from  low-to-high  implemented  at  AITS-JPO  to  obtain  PATROL® 
monitoring  and  intrusion  detection  system  events  to  a  top-level  enterprise 
manager  on  the  high  side. 

•  PATROL®  Explorer  maps  generated  for  AITS-JPO  and  USPACOM  TCCC. 

•  PATROL®  knowledge  module  developed  to  monitor  DCTS  (CuSeeMe)  at  AITS- 
JPO. 

•  PATROL®  integration  with  various  network  managers  and  trouble  ticket  systems 
provides  an  excellent  capability  for  enterprise  management  without  requiring 
installing  it  on  the  entire  enterprise. 
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Remaining  Challenges 

•  PATROL®  visualizations  are  not  easily  generated  (side  of  the  PATROL® 
console). 

•  PATROL®  Enterprise  Manager  maps  are  not  compatible  with  PATROL® 

Explorer  maps  (vector  vs.  raster). 

•  PATROL®  Explorer  raster  maps  do  not  have  sufficient  resolution  as  opposed  to 
PATROL  Enterprise  Manager  vector  maps. 

•  PATROL®  service  reporting  ideally  requires  separate  hardware  to  collect 
reportable  statistics  over  a  30-day  period. 

•  Training  is  definitely  required  to  install,  operate,  and  configure  the  product(s). 

•  Cooperation  (e.g.,  security  pennission)  with  the  various  stakeholders  is  a  must 
especially  where  no  single  organization  controls  the  network,  servers,  and 
applications. 

•  PATROL®  reports  a  number  of  valuable  network  and  application  events  out  of  the 
box,  however  it  must  be  tweaked  for  threshold  settings  and  generally  takes  about 

1  to  6  months  (depending  on  the  installation)  to  configure  it  properly  for  the 
enterprise. 

•  Until  an  interface  between  the  PATROL®  Enterprise  Manager  and  NMCI  is 
developed,  the  TCCC  will  not  be  able  to  monitor  the  complete  enterprise. 

BMC  Patrol  Summary 

BMC  patrol  provides  a  comprehensive  set  of  network  management  tools.  These  tools 
give  network  managers  the  ability  to  interact  with  other  management  tools,  thus 
providing  the  user  with  a  needed  capability  and  supporting  a  Green  Suitability  rating. 
Difficulties  generating  visualizations  and  displaying  vector  and  raster  maps  lead  to  a 
YELLOW  rating  for  Usability.  The  technical  rating  is  also  YELLOW  due  to  the  installation 
and  configuration  demands  and  instabilities. 


Suitability 

Usability 

Technical 

BMC  Patrol 

© 

A 

A 

Special  Projects 


The  special  projects  were  single  attempts  to  develop  a  tool  or  application  for  a  specific  use.  Most 
of  these  projects  were  hardware  or  software  that  did  not  lend  themselves  to  any  meaningful 
assessment  other  than  proper  installation  and  operation.  Some  of  these  special  projects,  however, 
were  assessed  and  include  the  following: 

•  USSTRATCOM  Portal 

•  JWID  Administrative  Portal 


•  Quality  of  Service  (QoS) 


91 


Joint  Military  Utility  Assessment 


•  Time-Step  Virtual  Private  Network  (VPN) 

•  Remote  Routing  Access  Server  (RRAS)  VPNs  with  Public  Key  Infrastructure  (PKI) 

•  3Com®  Embedded  Firewall  (EFW)  NIC  Cards 

•  Internet  Protocol  version  6  (IPpv6) 

•  NetlQ 

•  Net  VCR 

•  Multi-Router  Traffic  Graphing  (MRTG)  tool 
USSTRA  TCOM  Portal 

Assessment  Events 

•  USSTRATCOM  Consequence  Management/Response  Demonstration  (Oct  02) 
(Appendix  I) 

Senior  decision-makers  were  guided  through  an  exercise,  and  then  feedback  was 
collected  on  how  the  CINC  2 1  technologies  (including  the  C2  Portal)  were  perceived 
to  support  work  processes  in  Consequence  Management/Response.  In  addition, 
human  factors  professionals  performed  heuristic  evaluations  of  the  user  interface. 

Suitability 

Positive  Findings 

Al.  The  accuracy  of  situation  awareness  and  understanding  among  decision¬ 
makers  was  increased. 

B 1 .  The  time  needed  to  collect,  identify,  and  integrate  information  was  reduced. 

B3.  The  time  to  display  requested  information  decreased,  so  that  additional  time  is 
available  for  analyzing  decision  options. 

HI.  The  currency  of  information  was  sufficient  for  the  decision-maker’s 
requirements. 

H2.  The  completeness  of  the  information  was  sufficient  for  decision-makers' 
requirements. 

H6.  The  amount  of  relevant  data  available  within  the  decision  cycle  increased. 

G2.  Decision  choices  for  crisis  response  were  framed  faster. 

18.  CINC  21  technologies  were  accepted  by  the  target  users. 

Remaining  Challenges 

H3.  The  precision  of  information  was  not  sufficient  for  decision-makers 
requirements. 

Usability 

Positive  Findings 

A2.  Ability  to  scale  and  tailor  visualization  displays  increased. 
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D2.  The  number  of  steps  or  procedures  used  to  exchange  information  was 
reduced. 

El.  The  number  of  actions  needed  to  coordinate  between  participants  was 
reduced. 

13.  Workload  was  reduced. 

15.  The  number  of  staff  needed  to  perform  the  task  was  reduced. 

17.  Watchstanders  were  able  to  relay  infonnation  faster  when  managing  routine 
theater  activity. 

J2.  Conveyance  of  information  on  the  display  device  platforms  was  supported. 

J3.  The  information  was  presented  in  a  form  that  was  usable  by  all  distributed 
members. 

LI.  The  time  was  reduced  in  establishing  a  collaborative  framework. 
Remaining  Challenges 

-NONE- 

Technical 

Positive  Findings 

A3.  The  decision  support  and  knowledge  management  was  advanced. 

Jl.  Information  transmittal  attempts  were  successful. 

Remaining  Challenges 

H2.  The  completeness  of  the  information  was  not  sufficient 

K5.  The  system  did  not  provide  a  secure  means  for  exchanging  information 
among  distributed  users. 

General  Findings 

Positive  Findings 

•  The  portal  provides  current  information  to  all  users. 

•  It  decreases  the  number  of  steps  and  staff  needed  to  develop  and  exchange 
information. 

•  Target  users  accepted  the  portal. 

Remaining  Challenges 

•  The  portal  is  unable  to  exchange  information  across  security  levels. 

•  The  number  of  steps  required  to  drill  down  for  more  detail  needs  to  be 
reduced. 

•  An  improved  human-computer  interface  is  needed  to  enhance  usability. 

•  There  are  unresolved  procedural  issues  regarding  dynamic  updating. 
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•  The  completeness  of  information  was  not  sufficient  for  decision-makers’ 
requirements  (still  in  development). 

USSTRATCOM  Portal  Summary 

During  the  USSTRATCOM  October  2002  demonstration,  senior  decision-makers  said 
that  the  C2  Portal  was  a  valued  addition  to  their  current  process.  Suitability  thus  receives 
a  Green  rating. 

Usability  receives  a  Green  rating,  although  some  human-computer  interface 
improvements  are  needed.  Also,  care  must  be  taken  in  limiting  the  number  of  levels  a 
user  will  have  to  access  to  find  information  while  in  the  middle  of  giving  a  briefing. 

As  for  the  Technical  category,  a  YELLOW  rating  is  assigned.  At  the  time  of  the 
assessment,  the  portal  was  not  fully  functional.  The  portal  was  unable  to  exchange 
information  across  security  levels.  The  composite  ratings  for  the  USSTRATCOM  Portal 
are  shown  below. 


Suitability 

Usability 

Technical 

USSTRATCOM  Portal 

© 

© 

A 

JWID  Administrative  Portal 
Assessment  Events 

•  Joint  Warrior  Interoperability  Demonstration  (May  02)  (Appendix  U) 

•  HCI  evaluation  (May  02)  (Appendix  Y) 

Data  were  collected  from  distributed  operational  users  in  four  coalition  countries 
(Australia,  Canada,  the  United  Kingdom  and  the  United  States)  as  they  used  the 
JWID  Administrative  Portal  to  perform  scenario  tasks  during  the  Joint  Warrior 
Interoperability  Demonstration  in  May  2002.  Users  also  completed  questionnaire 
items  that  addressed  various  issues  concerning  functional  utility  and  interface 
usability.  In  addition,  human  factors  professionals  performed  a  heuristic 
evaluation  of  the  user  interface  in  a  laboratory  environment. 

Suitability 

Positive  Findings 

Al.  The  accuracy  of  situation  awareness  and  understanding  among  decision¬ 
makers  was  increased. 

B 1 .  The  time  needed  to  collect,  identify,  and  integrate  information  was  reduced. 

B3.  The  time  to  display  requested  information  was  decreased,  so  that  additional 
time  was  available  for  analyzing  decision  options. 

Cl.  Planners’  satisfaction  with  the  decision  was  increased. 

E3.  Synchronization  of  information  management  among  key  players  was 
increased. 
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F2.  The  sharing  of  inputs  among  the  decision-makers  was  increased. 

F3.  The  ability  to  generate  alternative  CO  As  was  improved. 

18.  Target  users  accepted  CINC  21  technologies. 

Remaining  Challenges 

G2.  Decision  choices  for  crisis  response  were  not  framed  faster. 

Usability 

Positive  Findings 

J3.  Information  was  presented  in  a  fonn  that  was  usable  by  all  distributed 
members. 

Remaining  Challenges 

D2.  The  number  of  steps  or  procedures  used  to  exchange  information  was  not 
reduced. 

16.  CINC  21  technologies  did  not  facilitate  mission-relevant  training. 

J2.  Conveyance  of  information  on  the  display  device  platforms  was  not  well- 
supported. 

LI .  Time  was  not  reduced  in  establishing  a  collaborative  framework  (display  of 
data  for  collaborative  efforts). 

Technical 

Positive  Findings 

A3.  Decision  support  and  knowledge  management  was  advanced. 

K2.  The  controllers  had  central  access  to  coalition  non-combatant  evacuation 
operation  information. 

Remaining  Challenges 

J 1 .  Infonnation  transmittal  attempts  often  were  not  successful. 

K3.  Increased  control  of  C2  infrastructure  did  not  provide  security  assurances 
across  multiple  networks,  databases,  and  applications. 

Ml.  Server  compatibility  was  a  problem. 

General  Findings 

Positive  Findings 

•  Coalition  partners  were  able  to  share  information  quickly. 

•  Coalition  partners  were  able  to  gain  situation  awareness  and  understanding 
quickly. 

•  Coalition  partners  were  able  to  synchronize  their  planning  efforts. 


Joint  Military  Utility  Assessment 


Remaining  Challenges 

•  The  capability  to  establish  and  maintain  stable  connectivity  is  needed. 

•  User  assistance  in  the  forms  of  embedded  training,  user  aids,  and  on-line  help 
is  currently  not  available. 

•  The  number  of  steps  required  to  manipulate  information  within  the  framework 
of  multiple  applications  is  excessive. 

•  The  user  interface  needs  to  be  designed  for  real  operational  use,  not  to  support 
specific  exercises. 

JWID  Administrative  Portal  Summary 

Users  endorsed  the  value  of  collaborative  planning  and  sharing  operational  planning 
information.  However,  the  JWID  Administrative  Portal  was  not  designed  as  a  transition- 
quality  product.  It  was  a  vehicle  to  integrate  (1)  diverse  applications  developed  by  the 
coalition  partners  and  (2)  the  JWID  scenario.  The  lack  of  ability  to  generalize  [SRB2]to 
other  settings  is  the  principal  reason  for  the  YELLOW  rating  for  Suitability. 

Multiple  modes  were  available  on  the  JWID  Administrative  Portal  for  users  to  access  the 
diverse  coalition  applications.  However,  the  location  of  these  applications  was  not  clearly 
outlined.  Some  web  parts  were  located  off  the  screen  with  no  indication  of  their  existence 
unless  the  user  scrolled  down  the  page.  There  was  no  online  user  assistance  and  little 
training.  Many  users  found  that  navigating  within  the  portal  and  manipulating 
information  on  the  portal  was  difficult.  Usability  is  rated  YELLOW. 

Establishing  and  maintaining  connectivity  was  a  problem  of  such  magnitude  that  users 
could  not  separate  the  system  support  issues  from  the  applications  themselves.  There 
were  repeated  disruptions  to  the  collaborative  sessions  and  the  sharing  of  information. 

The  Technical  rating  is  Red. 


Suitability 

Usability 

Technical 

JWID  Administrative  Portal 

A 

A 

« 

Quality  of  Service  (QoS) 

Quality  of  Service  (QoS)  is  a  set  of  capabilities  that  allow  you  to  create  differentiated 
services  for  network  traffic,  thereby  providing  better  service  for  selected  network  traffic. 

With  QoS,  you  can  increase  bandwidth  for  critical  traffic,  limit  bandwidth  for  non-critical 
traffic,  and  provide  consistent  network  response.  Implementing  QoS  in  your  network  can 
allow  the  efficient  use  of  expensive  network  connections.  Service  Level  Agreements  (SLAs) 
can  be  established  directly  with  the  customers  of  the  network. 

The  CINC  2 1  ACTD  collaboration  activities  require  the  ability  to  provide  acceptable  quality 
audio  (and  sometimes  video)  over  DoD  unclassified,  secret,  and  coalition  data  networks.  QoS 
mechanisms  based  on  the  Internet  Engineering  Task  Force  (IETF)  Differentiated  Services 
mechanisms  have  been  used  by  CINC  2 1  to  prioritize  important  collaboration  traffic  during 
demonstrations  and  exercises.  Packets  are  marked  for  prioritization  (e.g.,  critical,  flash,  etc.) 
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at  the  edges  of  the  network,  and  the  core  routing  devices  must  be  configured  to  honor  the 
markings.  The  experience  and  knowledge  gained  from  these  network  prioritization  activities 
has  been  passed  on  to  other  DISA  activities  and  plans  are  in  the  works  to  field  them  on  the 
operational  DoD  networks. 

Fieldings 

CINC  2 1  QOS  capabilities  have  been  fielded  in  various  configurations,  and  on  the  DISN- 

LES  Unclassified  and  Coalition  networks.  They  are  as  follows: 

•  October  2000  five  node  demonstration  of  CINC  2 1  support  of  collaboration  traffic 
with  QoS  for  the  USPACOM  Deputy  COCOM  (Figure  7).  Two  sites  in  Hawaii,  two 
sites  in  Virginia,  and  one  in  San  Diego  were  linked  by  TimeStep  VPNs  over  the 
Unclassified  but  sensitive  (N-level)  Internet  Protocol  Router  Network  (NIPRNET) 
and  DISN-LES  Unclassified  networks.  The  Cisco1'  QoS  Policy  Manager  was  used  to 
configure  Cisco  "  routers  in  the  network  to  prioritize  the  collaboration  traffic  over 
other  traffic  on  congested  links.  With  the  QoS  mechanisms  turned  on,  the  audio  and 
video  were  of  good  quality;  without  them  turned  on,  the  video  and  audio  quality  was 
very  poor. 

•  JWID’01  and  JWID’02  exercises  on  the  DISN-LES  Combined  Federation  Battle  Labs 
(CFBL)  network,  QoS  was  set  in  the  Cisco  "  routers  of  the  Combined  Federated 
Battlelab  Net  (CFBLNet)  to  prioritize  the  IP  Phone  traffic  used  to  coordinate  between 
sites  in  the  U.S.  and  participants  in  the  U.K.,  Canada,  Australia,  and  New  Zealand.  In 
JWID  ’02,  additional  work  was  done  to  prioritize  the  DCTS  traffic. 


Figure  7.  CINC  21  QoS  and  VPN  configuration  for  October  2000  demonstration. 

•  Multi-Protocol  Label  Switching  (MPLS)  QoS.  CINC  21  staff  worked  with  North 
Atlantic  Treaty  Organization  (NATO)  engineers  to  set  up  a  MPLS  VPN  with  QoS 
over  the  Unclassified  DISN-LES. 

Positive  Findings 

•  QoS  capabilities  were  demonstrated  in  various  demonstrations  and  exercises. 

•  QoS-enabled  networks  with  changing  policies  can  benefit  from  use  of  Cisco  " 
QoS  Policy  Manager  to  set  policies  on  many  routers  quickly. 
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Remaining  Challenges 

•  There  needs  to  be  business  rules/operational  procedures  to  decide  who  gets 
priority  on  operational  networks. 

•  Rules  are  needed  to  determine  what  gets  prioritized.  Prioritization  of  audio 
and  text  collaboration  is  the  first  choice;  video  should  only  be  prioritized  if  it 
is  critical. 

•  Password  access  to  the  routers  in  the  network  is  needed. 

•  More  “QoS-aware”  applications  are  needed. 

Recommendations  for  Future 

•  Test  results  showed  that  QoS  could  be  used  to  satisfy  Service-Level 
Agreements  with  network  customers. 

•  The  use  of  QoS  mechanisms  is  critical  to  supporting  multimedia  over  IP 
networks.  CINC  21  staff  have  provided  inputs  to  two  DISA  working  groups; 
one  doing  IP  QoS  evaluations  and  experiments,  and  the  other  doing  MPLS 
QoS  work.  It  is  expected  that  some  of  these  capabilities  will  be  fielded  on  the 
operational  NIPRNet  and  Secret  IP  Router  Network  (SIPRNet)  soon. 

QoS  Summary 

CINC  21  successfully  fielded  various  Differentiated  Services  based  on  QoS 
configurations  on  multiple  networks.  A  Cisco8'  QoS  Policy  Manager  was  delivered  to 
CINCPAC  for  their  further  use  and  evaluation.  The  capability  does  provide  flexibility  in 
network  traffic  management.  Thus,  Suitability  is  Green.  Not  enough  data  were  collected, 
however,  to  assess  the  Usability  or  Technical  classifications. 


Suitability 

Usability 

Technical 

Quality  of  Service 

© 

© 

© 

Virtual  Private  Network  (VPN) 

CINC  21  provides  protected  network  connectivity  between  users  at  multiple  sites,  and  in 
some  cases,  connects  individuals  working  from  home  or  a  hotel  to  a  CINC  21  server  site. 
VPNs  have  been  used  to  provide  this  environment,  using  Internet  Protocol  Security  (IPSec) 
encapsulation  and  encryption  to  protect  against  external  tampering  and  eavesdropping.  IPSec 
is  an  evolving  standard  from  the  IETF,  supporting  authentication,  privacy,  and  key 
management.  A  long-term  DoD  goal  is  to  use  IPSec  VPNs  to  create  Secret  and  above 
enclaves  over  unclassified  networks.  IPSec  is  not  NSA-Type  I-approved  at  this  time,  so  it  can 
be  used  to  provide  privacy,  but  cannot  replace  Tactical  FASTLANE1'  (TACLANE)/National 
Encryption  System  (NES)-type  packet  encryptors  for  U.S.  Secret  and  above  data. 

One  popular  use  of  IPSec-encrypted  VPNs  today  is  to  encapsulate  collaboration  traffic  such 
as  Microsoft  NetMeeting1' ,  which  is  blocked  by  most  firewalls  because  it  requires  numerous 
ports  to  be  opened.  The  encrypted  VPN  uses  only  one  port  and  has  the  added  benefit  of 
providing  privacy  to  the  collaboration  session. 
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When  the  CINC  21  VPN  activity  was  started,  there  was  only  one  FIPS  140-2  certified  VPN 
Gateway — the  Alcatel"  TimeStep  VPN  Gateway.  Later,  the  Cisco"  3000  series  VPN 
Concentrator  was  added  to  the  FIPS  140-2  certified  list.  NSA  strongly  recommended  that 
CINC  21  only  use  certified  devices.  In  addition  to  TimeStep,  CINC  21  looked  at  the 
Microsoft®  Remote  Routing  and  Access  Server  gateway  VPN. 

Fieldings 

CINC  21  VPNs  have  been  fielded  in  various  configurations  and  on  unclassified  and 
classified  networks  (see  Appendix  Z  for  details  regarding  the  RRAS  VPN).  Fielded 
VPNs  are  as  follows: 


•  October  2000  five-node  demonstration  of  CINC  2 1  VPNs  supporting  collaboration 
traffic  with  QoS  and  tunneled  IP  Multicast  for  the  USPACOM  Deputy  COCOM 
(Figure  7).  Two  sites  in  Hawaii,  two  sites  in  Virginia,  and  one  in  San  Diego  were 
linked  by  TimeStep  VPNs  over  the  NIPRNET  and  DISN-LES  unclassified  networks. 

•  VPN  between  USSTRATCOM  and  the  AITS-JPO  on  the  Secret  DISN-LES  and 
SIPRNET  using  TimeStep  VPN  Gateways.  This  configuration  provided  DCTS 
collaboration  capabilities  for  USSTRATCOM  users  interfacing  with  USPACOM  and 
AITS-JPO  users.  The  VPN  gateway  at  the  AITS-JPO  was  configured  to  support  split 
tunneling,  with  traffic  to/from  USSTRATCOM  IP  addresses  getting  encrypted/ 
unencrypted,  while  all  other  traffic  was  configured  to  pass  through  the  VPN  Gateway 
in  “clear”  unencrypted  mode.  This  configuration  allowed  users  in  USPACOM 
without  VPN  capability  to  connect  to  a  collaboration  server  at  the  AITS-JPO  that  was 
linked  with  USSTRATCOM,  supporting  collaboration  sessions  with  all  sites.  The 
diagram  below  (Figure  8)  shows  the  basic  configuration. 


DTRA  DCTS 
Client 


CINC  21 


Figure  8.  CINC  21  VPN  configuration  between  USSTRATCOM  and  the  AITS-JPO. 
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•  VPNs  supported  the  CINC  2 1  unclassified  development  environment  at  MITRE. 
TimeStep  VPN  gateways  were  fielded  at  MITRE  and  the  AITS-JPO  on  the 
unclassified  DISN-LES  to  support  CINC  2 1  development  efforts.  This  configuration 
required  the  ability  to  allow  remote  users  to  connect  from  the  Internet  using  client 
VPN  software.  The  TimeStep  VPN  client  was  unsatisfactory  for  this  purpose.  It  was 
difficult  to  load  on  users’  personal  computers  and  most  difficult  to  remove — usually 
requiring  a  complete  rebuild  of  the  system. 

CINC  21  fielded  the  Microsoft®  RRAS  at  MITRE  to  provide  better  VPN  client 
support.  The  RRAS  server  worked  with  the  VPN  client  that  comes  with  Windows® 
2000,  strict  lockdown  procedures  were  performed  on  the  RRAS  server,  and  users 
were  given  PKI  certificates  for  user  authentication.  Appendix  Z  is  a  report  from  the 
Joint  Information  Operations  Center,  Infonnation  Warfare  Analysis  Division  on  a 
security  analysis  of  the  Microsoft®  RRAS/Alcatel®  Permit  Timestep  4600  VPN 
Gateway  combination  solution  and  the  Cisco®  VPN  concentrator/Microsoft5'  Internet 
Authentication  Service  (IAS),  Remote  Access  Dial-In  Service  (RADIUS), 
combination  solution. 

•  Multi-Protocol  Label  Switching  (MPLS)  VPN.  CINC  2 1  staff  worked  with  NATO 
engineers  to  set  up  a  MPLS  VPN  over  the  Unclassified  DISN-LES.  This  type  of  VPN 
provides  encapsulation  (not  encryption)  and  routing  isolation.  DISA  is  interested  in 
this  configuration  for  the  future. 

•  Cisco®  3030  and  3005  VPN  Concentrators.  When  Cisco®  was  added  to  the  FIPS 
140-2  certified  list,  CINC  21  staff  evaluated  the  product  for  potential  future  use.  The 
Cisco  8  3000  series  VPN  Concentrators  function  well  as  VPN  gateways,  and  the 
Cisco®  client  is  easy  to  load  and  unload  from  user  personal  computers,  unlike  the 
TimeStep  client. 

Positive  Findings 

•  VPNs  tunneling  through  firewalls  for  collaboration.  The  unclassified  networks 
have  many  firewalls  that  otherwise  block  the  collaboration  tools  that  use 
dynamic  (changing)  ports.  VPNs  encapsulate  this  traffic  into  one  port. 
Hardware  acceleration  built  into  the  TimeStep  and  Cisco®  VPN  gateways 
allow  for  acceptable  performance  of  collaboration — otherwise  the  overhead  of 
encryption  can  cause  degraded  audio  and  video. 

•  Split  VPN  tunnel  for  USSTRATCOM — connecting  users  that  could  not 
otherwise  collaborate  over  the  SIPRNET. 

•  Class/QoS  using  Differentiated  Services  capabilities  for  VPNs. 

•  Internet  Protocol  (IP)  Multicast  tunnels  through  VPNs. 

•  Use  of  PKI  for  user  authentication. 

Remaining  Challenges 

•  TimeStep  VPN  client  software  is  judged  as  not  user/PC  friendly. 

•  PKI  authentication  is  good  to  do,  but  key  management  is  difficult  and  time 
consuming. 
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•  Microsoft®  RRAS  is  the  most  convenient  VPN  gateway  for  users,  but  it  is  too 
vulnerable  to  attack  because  it  runs  the  Microsoft®  Operating  System.  The 
TimeStep  and  Cisco®  VPN  Gateways  are  dedicated  boxes  and  less  vulnerable 
to  hackers. 

•  Remote  clients  connecting  through  the  RRAS  server  sometimes  experienced 
excessive  delays/timeouts  that  were  due  to  Domain  Name  Service  (DNS)  and 
WINS  problems. 

•  Scaling  issue:  Management  of  multiple  VPNs  difficult — each  VPN  is  a 
different  network  and  you  need  connectivity  and  management/monitoring 
tools  on  each. 

Recommendations  for  Future 

•  In  the  future,  significant  cost  savings  could  be  obtained  through  the  use  of 
NSA  Type-l-certified,  IPSec-based  VPNs.  This  configuration  could  become  a 
reality  within  the  next  year. 

•  Use  Cisco®  VPNs  for  the  near  term — they  are  FIPS-140-2-certified  and  have 
user-friendly  client  software. 

VPN  Summary 

CINC  2 1  successfully  fielded  various  VPN  configurations  on  multiple  networks.  The 
VPN  tools  do  offer  some  capability,  but  they  do  not  appear  well  matched  for  the 
warfighter  needs.  Suitability,  then,  is  rated  YELLOW.  Not  enough  data  were  collected, 
however,  to  assess  the  Usability  or  Technical  classifications.  It  should  be  noted  that 
VPNs  are  considered  to  be  the  best  route  for  providing  the  necessary  security  within 
networks.  Future  efforts  in  VPN  are  expected  to  provide  this  capability  in  a  reliable  and 
usable  form  to  all  warfighters. 


Suitability 

Usability 

Technical 

VPN 

A 

3Com ®  Embedded  Firewall  (EFW)  NIC  Cards 

Protecting  against  insider  threats  is  an  issue  of  concern  to  the  CINC  21  ACTD.  VPN 
gateways  and  firewalls  are  placed  at  the  entrance  to  sites  to  protect  against  outsider  threats, 
but  little  is  normally  done  to  protect  against  insider  threats.  The  3COM®  Embedded  Firewall 
product  was  evaluated  by  CINC  21  and  used  in  JWID’02  to  detect  and  block  a  Denial  of 
Service  attack.  The  EFW  capability  is  built  into  commercially  available  Ethernet  NIC  cards 
and  managed  from  a  management  station  running  the  3COM  EFW  Policy  Manager.  A 
vulnerability  assessment  of  the  3COM1'  EFW  was  funded  by  CINC  21  and  performed  by  the 
Joint  Information  Operations  Center  Branch  (JIOC)  Technology  Evaluation. 

The  3Com®  EFW  is  a  product  and  technology  that  prevents  insider  attacks  and  outside  hacks 
by  placing  a  low-cost,  easy-to-use  network  access  control  point  directly  in  front  of  each 
critical  host  on  the  network.  The  EFW  network  access  control  point  is  independent  of  the 
host  and  controls  all  accesses  to  and  from  the  host.  This  independence  means  it  protects  the 
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network  from  attacks  launched  from  the  host  itself,  as  well  as  protecting  the  host  from 
network  attacks.  It  also  means  a  user  or  malicious  code  that  gains  access  to  administrator 
mode  on  the  host  cannot  modify  the  configuration  of  the  access  control  point.  All  the  EFW 
access  control  points  in  an  enterprise  are  centrally  managed,  and  each  has  a  simple,  easy-to- 
understand  security  policy.  These  are  as  follows: 

•  56-  to  70-MB  data  throughput  increase  with  encryption  offloading  to  the  card 

•  EFW  is  stronger  than  host/application  security  because  it  is  independent  of  the 
existing  host  OS 

•  Survivable  security  solution  (NIC  is  very  secure  in  itself) 

•  Provides  fiber  support 

•  Provides  thin-client  support 

•  Intrusion  detection  capabilities  (can  detect  when  unauthorized  user  is  using  policy 
server  and  shut  application  down) 

The  EFW  has  the  three  properties  that  are  the  focus  point  for  security: 

•  It  is  tamper-proof  in  that  the  policy  and  enforcement  mechanism  can  only  be  changed 
by  authorized  administrators. 

•  It  cannot  be  bypassed,  so  the  only  way  to  go  between  the  network  and  the  host  is 
through  an  EFW  access  control  point. 

•  The  policies  are  simple  enough  to  be  analyzed  by  a  security  auditor  who  wants  to 
ensure  the  system  is  enforcing  an  appropriate  security  policy. 

The  EFW  system  architecture  consists  of  a  centralized  Policy  Server  that  distributes  access 
control  policies  to  its  slaves.  The  slaves  are  3Com®  3CR990  NICs  on  the  hosts  protected  by 
the  EFW.  The  3CR990  was  designed  in  collaboration  with  Microsoft®  for  optimal  IPSec 
VPN  performance  on  Windows®  2000  and  Windows®  XP  hosts.  It  has  an  onboard  encryption 
chip  for  IPSec  encryption  as  well  as  a  RISC  processor. 

Secure  Computing  has  developed  new  security  firmware  that  runs  on  the  Reduced  Instruction 
Set  Computer  (RISC)  processor  and  converts  the  NIC  into  an  EFW  component.  This 
firmware  converts  the  NIC  into  a  security  slave  of  the  central  Policy  Server,  also  developed 
by  Secure  Computing.  That  is,  the  NIC  only  accepts  policy  configuration  information  that 
comes  over  an  encrypted  communication  channel  from  the  Policy  Server. 

Managed  by  the  Policy  Server,  each  NIC  enforces  a  firewall  policy  that  filters  access  to  the 
host  based  on  source  and  destination  IP  addresses  and  ports.  It  also  filters  based  on  whether 
or  not  its  protected  host  initiates  the  session.  Thus,  it  controls  what  services  the  protected 
host  provides  to  each  remote  host. 


Fieldings 

The  EFW  NICs  were  fielded  at  the  AITS-JPO  on  the  Unclassified  DISN-LES  for  testing 
along  with  an  installation  at  USPACOM  during  JWID.  During  JWID,  the  cards  were 
used  to  demonstrate  how  denial  of  service  could  be  managed  and  blocked  once  it  was 
detected. 
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Positive  Findings 

(r) 

•  Basic  functionality.  The  3Com  EFW  NIC  was  tested  and  verified  (i.e., 
ability  to  shutdown  specific  ports  and  services  such  as  ftp  and  telnet). 

•  The  Policy  Server  distributed  policies  to  Client  NIC  cards  without  delay  or 
utilizing  much  of  the  system  resources. 

•  Firewall  functionality  is  independent  of  the  host  Operating  system. 

•  Successful  block  of  malicious  traffic  during  JWID. 

Remaining  Challenges 

•  NIC  cards  are  easily  rendered  inoperable. 

•  Virtual  Private  Group  capability — the  ability  to  have  VPNs  among  groups  not 
just  peer  to  peer.  This  capability  was  promised,  but  not  available  on  the 
version  used  for  testing. 

•  Failed  attempts  in  connecting  clients  at  multiple  sites. 

•  Tested  version  of  the  Policy  Server  did  not  support  Ipsec  with  Windows® 
2000  VPN  capability.  This  function  would  be  available  in  a  later  version. 


EFW  Summary 

There  was  little  opportunity  to  assess  this  capability,  i.e.,  not  enough  data  were  collected 
to  assess  any  of  the  mission  effectiveness  criteria. 


Suitability 

Usability 

Technical 

EFW 

Other  Special  Projects 

Several  other  Special  Projects  were  also  initiated.  Technical  assessments  were  made  on  the 
following  capabilities: 

•  Internet  Protocol  version  6  (IPpv6) 

•  NetlQ 

•  Net  VCR 

•  MRTG  tool 

Internet  Protocol  version  6  (IPv6) 

IPv6  is  an  emerging  standard  that  will  have  a  big  impact  on  the  DoD  in  5  to  10  years. 
CINC  21  collaborated  with  SSC  Charleston  to  build  a  native  IPv6  testbed  that 
communicates  over  the  Unclassified  DISN-FES  (heading  Edge  Services)  network.  The 
JIOC  perfonned  a  security  evaluation  of  IPv6.  The  testing  showed  that  not  all  the 
necessary  capabilities  are  complete  and  hardening  of  the  technology  is  needed.  IPv6  is 
rated  Yellow  for  Technical  because  there  is  still  significant  work  to  be  done  with  the 
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protocol  before  it  can  be  fielded  widely.  There  is  not  enough  information  to  rate 
Suitability  and  Usability. 


Suitability 

Usability 

Technical 

IPv6 

© 

A 

NetlQ 

NetlQ  is  a  network  performance  monitoring  tool  used  by  CIN  C21  during  JWID  ’02.  It  is 
very  good  at  measuring  application  and  server  performance  and  network  bandwidth 
availability  and  is  included  in  the  recommended  set  of  recommended  tools  in  CINC  21.  It 
is  rated  Green  for  Technical  because  it  is  a  commercial  off-the  shelf  product  that  provides 
critical  network  information.  There  is  not  enough  information  to  rate  Suitability  and 
Usability. 


Suitability 

Usability 

Technical 

NetlQ 

© 

© 

NetVCR 

NetVCR  was  used  in  JWID  ’02  to  provide  detailed  performance  measurements  and 
playback  capability  of  traffic  to  critical  CINC  2 1  servers.  It  is  included  in  the  set  of 
recommended  CINC  2 1  tools  because  it  provides  detailed  information  on  critical  parts  of 
the  network  infrastructure.  It  is  rated  Green  for  Technical,  but  it  is  a  costly  item  and  is 
normally  needed  at  multiple  locations.  There  is  not  enough  information  to  rate  Suitability 
and  Usability. 


Suitability 

Usability 

Technical 

NetVCR 

© 

© 

Multi-Router  Traffic  Graphing  (MRTG)  Tool 

The  MRTG  tool  was  used  in  JWID  ’00,  JWID  ’01,  and  JWID  ’02  by  CINC  21.  It  is  a 
freeware  tool  that  collects  and  graphs  aggregate  traffic  on  router  and  switch  ports.  It  is 
included  in  the  set  of  recommended  CINC  21  tools.  It  is  rated  Yellow  for  Technical 
because  many  of  the  DoD  networks  block  the  SNMP  (Simple  Network  Management 
Protocol)  traffic  that  MRTG  requires  to  provide  status.  There  is  not  enough  information 
to  rate  Suitability  and  Usability. 


Suitability 

Usability 

Technical 

MRTG 

© 

A 
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CINC  21  Prioritized  Requirements 


The  CINC  21  Operational  Manager  identified  44  system  requirements  for  advanced  information 
technology.  These  requirements  help  to  specify  the  four  COIs  in  greater  detail.  USPACOM  staff 
then  prioritized  these  requirements.  CINC  2 1  technology  focused  on  the  top  27  priority 
requirements.  The  summary  of  assessment  ratings  of  the  complete  set  of  prioritized  requirements 
is  shown  in  Table  1 1  along  with  the  applicable  COI  and  CINC  21  technology. 
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Table  1 1 .  Summary  of  prioritized  requirements. 


Priority 

System  Requirement 

COI 

Technology 

Assessment 

1 

Tdetailsl 

Develop  the  standard  desktop  client 
workstation  configuration  and  software 
to  allow  command  /  staff  organic 
support  (CINC  21 -configured  desktop 
workstation). 

1  -  Decision 
Focused 
Visualization 

EWS 

(not  assessed  directly) 

© 

2 

Tdetailsl 

Provide  domain-independent 
mechanisms  for  dynamically  tailoring 
the  presentation  of  information  to  a 
decision-maker  based  upon  their 
activities. 

1  -  Decision 
Focused 
Visualization 

DFC2  COA  Matrix 

FBV  AOR  Basing 

TCCC  Conseq.  Mgt. 

A 

3 

Tdetailsl 

Provide  domain-independent 
mechanisms  for  displaying  linkages 
between  related  events  /  items  both 
within  and  across  windows,  including 
drill-down. 

1  -  Decision 
Focused 
Visualization 

DFC2  COA  Matrix 

FBV  AOR  Basing 
Message  Tracker 

A 

4 

Tdetailsl 

Provide  generalized  XML-capable 
browser  and  integrate  with  the  KM 
database. 

1  -  Decision 
Focused 
Visualization 

DFC2 

STRATCOM  Portal 

A 

5 

Tdetailsl 

Provide  open,  extensible  geo-situation 
software  that  allows  acceptance  of  data 
and  geo-rendering  with  complete 
filtering,  labeling,  and  overlay  control. 

1  -  Decision 
Focused 
Visualization 

XIS  provided  as  part 
of  AOR  Basing 

(not  assessed 
directly) 

© 

6 

Tdetailsl 

Provide  the  standard  distributed 
collaboration  software  client 
configuration  and  software  to  allow 
command  /  staff  organic  support  to 
users. 

3  -  Collaboration 

DOTS 

A 

7 

Tdetailsl 

Develop  a  thematic  collaboration 
system  that  processes  information, 
produces  summaries  and  links  it  to 
other  information  articles,  allows 
analysis  and  collaboration. 

3  -  Collaboration 

DFC2 

A 

8 

Tdetailsl 

Develop  a  solution  for  secure 
authentication  on  SIPRNET  among  the 
COCOM’s  staff,  JTFs,  and 

Components. 

Not  addressed 
sufficiently  to 
allow 

assessment. 

4  -  Security 

User  Management 

® 

9 

Tdetailsl 

Provide  tools  for  monitoring  and 
visualizing  information  flows  across  the 
theater  based  upon  content,  priority, 
and  purpose. 

4  -  Enterprise 
Awareness 

DFC2  COA  Matrix 

FBV  AOR  Basing 

A 
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Priority 

System  Requirement 

COI 

Technology 

Assessment 

10 

Provide  tools  for  constructing  and 

not  addressed 

viewing  the  Information  Assurance 

Tdetailsl 

COP. 

11 

Provide  unclassified  restricted  (e.g., 

3  -  Collaboration 

DCTS 

NIPRNET)  conference  server  to  support 

A 

Tdetailsl 

multiple  distributed  collaboration 

sessions  internal  to  DoD. 

12 

Develop  a  secure  mechanism  that 

4  -  Security 

DCTS 

enables  collaboration  tools  to  work  in 

IWS 

(g) 

Tdetailsl 

the  presence  of  firewalls. 

VPN 

13 

Provide  a  secret  conference  server  to 

3  -  Collaboration 

DCTS 

A 

support  multiple  distributed 

IWS 

Tdetailsl 

collaboration  sessions  with  US  units. 

14 

Provide  unclassified,  unrestricted 

3  -  Collaboration 

DCTS 

conference  server  to  support  multiple 

A 

Tdetailsl 

distributed  collaboration  sessions  with 

organizations  and  agencies  external  to 

US  DoD. 

15 

Field  collaboration  tools  to  the  COCOM, 

3  -  Collaboration 

DCTS 

JTF,  and  Component  Commanders  that 

IWS 

A 

Tdetailsl 

provide  VTC,  voice,  instant  messaging, 

shared  virtual  workspace,  shared 

whiteboard,  and  shared  applications. 

16 

Implement  a  solution  to  achieve  run- 

Not  addressed 

Collaboration  tools 

time  interoperability  among  different 

sufficiently  to  allow 

Tdetailsl 

collaboration  tools. 

assessment. 

3  -  Collaboration 

17 

Provide  mechanisms  for  automatically 

3  -  Collaboration 

DFC2 

setting  up  and  managing  collaboration 

A 

Tdetailsl 

sessions  based  upon  workflow  and 

ZjlA 

operational  priorities. 

18 

Develop  a  solution  for  securely 

4  -  Security 

VPN 

connecting  the  COCOM  to  his  Allied 

Tdetailsl 

counterparts  over  unclassified 

W 

networks. 

19 

Provide  secret  conference  server  to 

3  -  Collaboration 

DCTS 

A 

support  multiple  distributed 

Tdetailsl 

collaboration  sessions  with  allies. 

20 

Provide  mechanisms  for  automating 

2  -  Knowledge 

Enterprise  C2 

staging  and  distribution  of  knowledge 

Management 

Infrastructure,  DFC2, 

Tdetailsl 

objects  throughout  the  theater 

COA  Matrix  and 

w 

(integrate  with  IDM). 

COINS 
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Priority 

System  Requirement 

COI 

Technology 

Assessment 

21 

Tdetailsl 

Provide  the  semantic  constructs 
required  for  knowledge  exchange  for 
critical  COCOM/JTF  warfighting  and 
theater  engagement  processes. 

2  -  Knowledge 
Management 

DFC2 

A 

22 

Tdetailsl 

Provide  tools  that  enable  structured  KM 
publishing  processes  that  enable 
producers  to  publish  their  products  as 
XML-based  documents. 

2  -  Knowledge 
Management 

Enterprise  C2 
Infrastructure 

© 

23 

Tdetailsl 

Build  a  knowledge  portal  for  viewing 
and  navigating  through  the  COCOM’s 
Ops-lntel  Brief,  SITREPs,  and 

Electronic  Battlebook. 

1  -  Decision 
Focused 
Visualization 

DFC2 

JWID  Admin  Portal 

© 

24 

Tdetailsl 

Provide  knowledge  portals  for  the  JTF 
and  Allies  that  enable  easy  access  to 
COCOM-provided  support  services. 

1  -  Decision 
Focused 
Visualization 

DFC2 

JWID  Portal 

A 

25 

Tdetailsl 

Implement  data  extraction  and 
automated  assembly  mechanisms  for 
automating  significant  portions  of  the 
COCOM’s  morning  brief. 

2  -  Knowledge 
Management 

DFC2  COA  Matrix 

RFI  Conseq.  Mgt. 

TCCC  Msg.  Tracker 

A 

26 

Tdetailsl 

Develop  automated  brief  preparation 
capability  based  upon  content  and 
update  of  the  information  /  knowledge 
base. 

1  -  Decision 
Focused 
Visualization 

DFC2 

Consequence  Mgmt. 

A 

27 

Tdetailsl 

Develop  consequence  management 
and  targeted  briefing  process. 

1  -  Decision 
Focused 
Visualization 

DFC2 

Consequence  Mgmt. 

A 

28 

Tdetailsl 

Develop  the  standard  conference  room 
configuration  to  allow  command  /  staff 
organic  support  (CINC  21 -configured 
conference  room). 

3  -  Collaboration 

DCTS 

IWS 

© 

29 

Tdetailsl 

Develop  DTDs  for  OPLANs  and 
CONPLANs. 

1  -  Decision 
Focused 
Visualization 

Enterprise  C2 
Infrastructure; 
Knowledge  Core  and 
DFC2:  Decision  Space 
Data  Schema,  Schema 
Management 

A 

30 

Tdetailsl 

Develop  the  dynamic  TPFDD. 

Not  addressed 

© 

31 

Tdetailsl 

Implement  mechanisms  for  automating 
processes  consistent  with  the  workflow 
management  system  currently  under 
development  at  PACOM. 

1  -  Decision 
Focused 
Visualization 

Task  Management 
System  (TMS) 

© 
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Priority 

System  Requirement 

COI 

Technology 

Assessment 

32 

Provide  next  generation  situation 

4  -  Enterprise 

DFC2  COA  Matrix 

Tdetailsl 

awareness,  planning,  and  force 
execution  tools  to  the  COCOM  and 
integrate  with  KM  environment. 

Awareness 

FBV  AOR  Basing 

A 

33 

Provide  the  infrastructure  that  enables 
selected  portions  of  the  COCOM’s 

Not  addressed 

Tdetailsl 

networks  to  support  prioritized  delivery 
of  products  and  bit-streams. 

34 

Tdetailsl 

Field  advanced  protocols  to  significantly 
reduce  traffic  latency  of  selected 
information  flows  through  theater 
networks. 

Not  addressed 

35 

Develop  an  accreditable  solution  for 
auto-generation  of  security  tags  for 

Not  addressed 

Tdetailsl 

archived  data  products  based  upon 
current  coalition  releasability  policy. 

36 

Provide  high-resolution,  large-screen 

1  -  Decision 

PACOM  Video  Wall 

displays  to  the  COCOM  that  are 

Focused 

STRATCOM  K-Wall 

© 

Tdetailsl 

capable  of  supporting  simultaneous 
crises  and  day-to-day  operations. 

Visualization 

37 

Develop  mechanisms  for  dynamically 

1  -  Decision 

PACOM  Video  Wall 

managing  the  pixel  real  estate  of  large 

Focused 

STRATCOM  K-Wall 

(Gj 

Tdetailsl 

displays  to  improve  comprehension. 

Visualization 

38 

Provide  alerting  mechanisms  that  auto- 

Not  addressed 

DFC2 

initiate  appropriate  visualization  events. 

sufficiently  to 

FBV 

Tdetailsl 

allow 

assessment. 

TCCC 

4  -  Enterprise 

Awareness 

39 

Implement  homogeneous  server-server 

4  -  Enterprise 

Enterprise  C2 

federations. 

Awareness 

Infrastructure; 

Tdetailsl 

Knowledge  Core  (XML 
Data  Services)  and 
DFC2  Schema 
management 

© 

40 

Develop  the  standard  mobile  client 
workstation  configuration  and  software 

Not  addressed 

Tdetailsl 

to  allow  command  /  staff  organic 
support  (CINC  21 -configured  mobile 
workstation). 

© 

41 

Provide  high-resolution,  large  displays 

4  -  Enterprise 

STRATCOM  K-Wall 

/ — X 

to  a  JTF  capable  of  supporting 

Awareness 

PV290 

(g) 

Tdetailsl 

simultaneous  crisis  operations. 
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Priority 

System  Requirement 

COI 

Technology 

Assessment 

42 

Tdetailsl 

Provide  a  method  for  accessing 
knowledge  objects  across  security 
enclaves  with  minimal  latency. 

Not  addressed 

43 

Tdetailsl 

Prepare  for  the  transition  from  Ipv4  to 

Ipv6  by  performing  selected 
experimentation  and  risk  reduction 
activities  on  selected  theater  networks. 

Not  addressed 

44 

Tdetailsl 

Provide  multi-modal  human-computer 
interface  capabilities  (e.g.  voice,  head¬ 
tracking,  gesture,  hand-pointing). 

Not  addressed 

The  following  section  provides  a  somewhat  expanded  discussion  of  the  basis  for  the  assessment 
of  each  CINC  2 1  requirement. 

Priority  Requirement  1  -  Develop  the  standard  desktop  client  workstation  configuration  and 
software  to  allow  command/staff  organic  support  (CINC  21 -configured  desktop  workstation). 
[COI-1] 

CINC  21  Technology  -  EWS  (not  assessed  directly) 

(G)  Assessment  -  A  standard  CINC  21  desktop  workstation  configuration  was  developed  and 
implemented  at  USPACOM.  The  EWS  configuration  included  a  standard  hardware  and 
software  suite  that  has  been  certified  for  use  in  operational  settings.  The  EWS  has  been 
used  during  several  exercises  and  continues  to  be  used  daily  for  routine  tasks.  The 
configuration  has  been  stable,  and  users  learned  to  use  the  component  tools  quickly. 
System  maintenance  and  upgrades  have  gone  smoothly  via  onsite  technical  support  staff 
and  system  administrators. 


Priority  Requirement  2  -  Provide  domain-independent  mechanisms  for  dynamically  tailoring  the 
presentation  of  information  to  a  decision-maker  based  upon  their  activities.  [COI-1] 

CINC  21  Technology  -  DFC2,  COA  Matrix,  FBV,  AOR  Basing,  TCCC,  CM 

A  Assessment  -  This  capability  has  not  been  fully  implemented  in  the  current  version  of 
DFC2  or  the  other  applicable  tools.  Several  of  them  do,  however,  provide  users  with 
limited  capability  to  manually  tailor  their  information  display  to  fit  their  needs.  This 
tailoring  is  not  yet  automatically  adjusted  based  on  their  activities. 


Priority  Requirement  3  -  Provide  domain-independent  mechanisms  for  displaying  linkages 
between  related  events/items,  both  within  and  across  windows,  including  drill-down.  [COI-1] 

CINC  21  Technology  -  DFC2,  COA  Matrix,  FBV,  AOR  Basing,  and  Message  Tracker 

A  Assessment  -  CINC  2 1  technologies  provide  limited  ability  to  display  linkages  between 
related  information.  They  also  provide  limited  capability  to  drill  down  to  get  additional 
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detail  as  desired.  Users  reported  that  these  capabilities  would  have  substantial  utility  for 
them.  Improvements  in  certain  aspects  of  the  user  interface  will  enhance  ease  of  use  and 
training. 


Priority  Requirement  4  -  Provide  generalized  XML-capable  browser,  and  integrate  it  with  the 
Knowledge  Management  database.  [COI-1] 

CINC  21  Technology  -  DFC2  Presentation  Layer  Portal,  USSTRATCOM  Portal 

A  Assessment-  Two  portal  environments  were  developed.  The  DFC2  was  based  on  Digital 
Dashboard  and  incorporated  fully  integrated  navigation,  table,  collaboration  and  timeline 
views  with  dynamic  XML-enabled  infonnation  access.  It  has  been  used  in  four 
USPACOM  leveraged  assessments  and  based  on  user  input  is  undergoing  upgrades  to  a 
more  powerful  Weblogic  portal  environment.  The  USSTRATCOM  portal  is  based  on  a 
more  conventional  ORACLE®  9i  portal  environment.  It  uses  dynamic  XML  enabled 
information  access.  It  is  operational  at  USSTRATCOM. 


Priority  Requirement  5  -  Provide  open,  extensible  geo-situation  software  that  allows  acceptance 
of  data  and  geo-rendering  with  complete  filtering,  labeling,  and  overlay  control.  [COI-1] 

CINC  21  Technology  -  XIS  (not  assessed  directly) 

(g)  Assessment-XIS  provides  web  based  geo-rendering  tools.  XIS  provides  XML  based 
input  and  output  capabilities  as  the  client  (Viewpoint)  and  Server.  The  XIS  server  tools 
form  the  basis  of  the  Enterprise  C2  Infrastructure  geo-spatial  XML  data  handling.  The 
XIS  server  functionality  is  used  for  all  exercises  as  part  of  the  Enterprise  C2 
Infrastructure.  The  XIS  Viewpoint  has  been  IOC  by  J4  for  AOR  Basing  Operational 
Package  information  access  and  display.  It  is  not  currently  used  due  to  lack  of  Data  Base 
Access/Content. 


Priority  Requirement  6  -  Provide  the  standard  distributed  collaboration  software  client 
configuration  and  software  to  allow  command/staff  organic  support  to  users.  [COI-3] 

CINC  21  Technology  -  DCTS 

Assessment  -  DCTS  was  provided  to  staff  as  the  standard  distributed  collaboration 
software  client  configuration.  Having  a  standard  software  toolset  made  it  easier  for  users 
to  collaborate  and  to  exchange  information.  Substantial  problems  with  DCTS  were 
experienced  over  many  different  assessment  events,  however.  Its  military  utility  was 
limited  by  the  degraded  performance  experienced  when  more  than  a  few  users  were 
connected  and  the  unstable  connectivity.  Collaboration  often  reverted  to  chat  and  other 
more  reliable,  low-bandwidth  tools  to  overcome  the  technical  problems  of  DCTS. 


Priority  Requirement  7  -  Develop  a  thematic  collaboration  system  that  ingests  infonnation, 
produces  summaries,  and  links  it  to  other  information  articles  to  allow  analysis  and 
collaboration.  [COI-3] 

Y  NC  21  Technology  -  DFC2  Assessment  -  Originally,  a  thematic  collaboration  system 
called  Corporate  Executive  Information  Systems  (CEIS)  was  envisioned  for  CINC  21  to  meet 
this  priority  requirement.  CEIS  was  not  developed,  however,  due  to  various  technical 
challenges.  Instead,  the  DFC2  concept  was  expanded  to  incorporate  much  of  this 


111 


Joint  Military  Utility  Assessment 


requirement.  The  DFC2  schema,  visible  to  users  via  the  Navigator,  enables  information  to  be 
organized  by  mission,  phase,  or  other  thematic  category.  Users  can  then  access  the  original 
information,  view  summaries,  and  see  links  to  other  related  information.  In  the  current  stage 
of  DFC2  development,  however,  these  functions  are  largely  done  manually  and  often  require 
a  great  deal  of  time  and  effort  by  operators. 


Priority  Requirement  8  -  Develop  a  solution  for  secure  authentication  on  SIPRNET  among  the 
COCOM’s  staff,  JTFs,  and  Components.  [COI-4] 


CINC  21  Technology  -  User  Management 


Assessment  -  This  requirement  was  not  sufficiently  addressed  to  allow  assessment.  User 
Management  capabilities  not  operationally  functional. 


Priority  Requirement  9  -  Provide  tools  for  monitoring  and  visualizing  information  flows  across 
the  theater  based  upon  content,  priority,  and  purpose.  [COI-4] 

CINC  21  Technology  -  DFC2,  COA  Matrix,  FBV,  AOR  Basing 

/yy  Assessment  -  Tools  have  been  created  that  afford  the  monitoring  and  visualizing  of 

information  flows.  However,  at  the  time  of  assessment  they  were  still  in  various  stages  of 
development. 


Priority  Requirement  10  -  Provide  tools  for  constructing  and  viewing  the  Information  Assurance 
COP. 


CINC  21  Technology  -  none 


Assessment  -  This  requirement  was  not  addressed  by  CINC  2 1 . 


Priority  Requirement  11  -  Provide  unclassified  restricted  (e.g.,  NIPRNET)  conference  server  to 
support  multiple  distributed  collaboration  sessions  internal  to  DoD.  [COI-3] 

CINC  21  Technology  -  DCTS 

/y\  Assessment  -  DCTS  did  enable  multiple,  distributed  DoD  groups  to  conduct  unclassified 
collaborative  sessions.  As  discussed  in  Requirement  6,  the  unstable  connectivity  and 
bandwidth  problems  degraded  the  utility  of  the  collaborative  sessions.  Users  needed  to 
rely  upon  chat  and  similar  techniques  and  were  often  unable  to  incorporate  audio  or 
video. 


Priority  Requirement  12  -  Develop  a  secure  mechanism  that  enables  collaboration  tools  to  work 
in  the  presence  of  firewalls.  [COI-4] 

CINC  21  Technology  -  DCTS,  IWS,  VPN 

Assessment  -  Video  teleconferencing,  text  chat,  and  other  collaboration  tools  were 
demonstrated  at  USPACOM  in  the  presence  of  firewalls.  VPN  techniques  enabled  these 
exchanges  to  operate  effectively.  QoS  techniques  were  demonstrated  to  significantly 
improve  signal  quality,  making  multi-modal  collaboration  more  viable. 

Priority  Requirement  13  -  Provide  a  secret  conference  server  to  support  multiple  distributed 
collaboration  sessions  with  U.S.  units.  [COI-3] 
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CINC21  Technology  -  DCTS,  IWS 

k  Assessment  -  DCTS  was  able  to  operate  on  the  SIPRNET  to  support  collaboration 
among  multiple  distributed  staff  within  and  across  commands.  As  noted  in  Requirement 
11,  connectivity  with  DCTS  was  fragile  and  the  quality  of  the  video  and  audio 
transmissions  degraded  significantly  as  more  participants  joined  the  conference. 
Connectivity  with  IWS  was  much  more  stable. 


Priority  Requirement  14  -  Provide  an  unclassified,  unrestricted  conference  server  to  support 
multiple  distributed  collaboration  sessions  with  organizations  and  agencies  external  to  U.S.  DoD. 
[COI-3] 

CINC  21  Technology  -  DCTS 

A  Assessment  -  Several  of  the  DCTS  tools  were  used  to  support  collaboration  among 
distributed,  multi-national  military  teams  during  the  JWID  exercise.  We  expect  that 
collaboration  with  non-military  groups  would  be  similar.  While  the  capability  to  connect 
these  groups  did  exist  via  a  conference  server,  the  quality  of  the  exchanges  was  fair  at 
best.  The  unstable  connectivity  that  was  experienced  further  reduced  the  military  utility 
of  this  technology.  Users  tended  to  rely  on  chat  and  other  more  reliable  tools. 


Priority  Requirement  15  -  Field  collaboration  tools  to  the  COCOM,  JTF,  and  Component 
Commanders  that  provide  video  teleconference  (VTC),  voice,  instant  messaging,  shared  virtual 
workspace,  shared  whiteboard,  and  shared  applications.  [COI-3] 

CINC  21  Technology  -  DCTS,  IWS 

A  Assessment-  Two  collaboration  toolsets  were  provided  along  with  two  additional  clients 
on  the  USPACOM  electronic  warfare  system  (EWS);  DCTS,  SAMETIME,  an  IWS 
client,  and  an  Odyssey  client.  Full  DCTS  suites  were  installed  at  USPACOM  and 
USSTRATCOM.  USPACOM  experienced  significant  performance  issues  with  DCTS 
and  did  not  make  the  suite  operational,  whereas  USSTRATCOM  is  using  DCTS 
operationally  within  the  command.  IWS  clients  were  provided  on  the  EWS  at 
USPACOM.  USPACOM  is  moving  to  make  IWS  operational.  Odyssey  was  provided  as  a 
client  on  EWS  but  was  not  used  by  either  command.  SAMETIME  clients  and  servers 
were  installed  at  USPACOM  only,  and  have  been  used  only  minimally. 


Priority  Requirement  16  -  Implement  a  solution  to  achieve  run-time  interoperability  among 
different  collaboration  tools.  [COI-3] 

CINC  21  Technology  -  Collaboration  tools 

i  \  Assessment  -  Not  addressed  sufficiently  to  allow  assessment.  Assessment  events  did  not 
sufficiently  address  cross-application  collaboration.  The  ability  to  transfer  data  between 
DCTS  and  IWS  was  demonstrated  at  USPACOM,  however. 


Priority  Requirement  17  -  Provide  mechanisms  for  automatically  setting  up  and  managing 
collaboration  sessions  based  upon  workflow  and  operational  priorities.  [COI-3] 

CINC  21  Technology  -  DFC2 
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A  Assessment  -  Automatic  setup  of  collaborative  chat  sessions  was  achieved  in  DFC2, 
based  on  users’  selection  of  operationally  defined  task  objects.  However,  workflow 
priorities  have  not  been  incorporated. 

Priority  Requirement  18  -  Develop  a  solution  for  securely  connecting  the  COCOM  to  Allied 
counterparts  over  unclassified  networks.  [COI-4] 

CINC  2 1  Technology  -  VPN 

©  Assessment  -  At  JWID,  use  of  an  unclassified  VPN  allowed  data  retrieval  by  coalition 
partners  located  in  four  nations  (Australia,  Canada,  United  Kingdom,  and  United  States) 
from  shared  databases. 


Priority  Requirement  19  -  Provide  a  Secret  conference  server  to  support  multiple  distributed 
collaboration  sessions  with  allies.  [COI-3] 

CINC  21  Technology  -  DCTS 

//y\x  Assessment  -  Several  DCTS  capabilities  were  used  to  support  collaboration  among 
distributed,  multi-national  groups  during  the  JWID  exercise.  While  the  capability  to 
connect  these  groups  did  exist  via  a  classified  conference  server,  the  quality  of  the 
exchanges  was  fair  at  best.  The  unstable  connectivity  that  was  experienced  further 
reduced  the  military  utility  of  this  technology.  Again,  users  tended  to  rely  on  chat  and 
other  more  reliable  tools. 


Priority  Requirement  20  -  Provide  mechanisms  for  automating  staging  and  distribution  of 
knowledge  objects  throughout  the  theater  (integrate  with  [IDM]).  [COI-2] 

CINC  21  Technology  -  Enterprise  C2  Infrastructure,  DFC2,  COA  Matrix  and  COINS 

©  Assessment  -  The  Enterprise  C2  Infrastructure  is  a  collection  of  technologies  that 

facilitate  access  to  information  throughout  the  theater.  The  hooks  to  IDM  are  available 
and  ongoing  discussions  are  underway  with  IDM  as  they  propagate  hardware  and 
software  into  the  COCOMs.  DFC2  provides  the  access  to  the  objects  across  the  theater. 
The  COA  matrix  and  the  COINS  interoperability  are  based  on  the  Enterprise  C2 
Infrastructure.  The  Enterprise  C2  Infrastructure  is  operational  on  the  SIPRNET.  COINS 
is  operational  on  COWAN  and  COA  Matrix  is  operational  at  USSTRATCOM. 


Priority  Requirement  21  -  Provide  the  semantic  constructs  required  for  knowledge  exchange  for 
critical  COCOM/JTF  warfighting  and  theater  engagement  processes.  [COI-2] 

CINC  21  Technology  -  DFC2 

/yv  Assessment  -  DFC2  organized  information  around  user-defined  themes,  such  as 

missions,  phases,  etc.  This  explicit  hierarchical  structure  provides  semantic  constructs 
that  can  facilitate  knowledge  exchange  between  COCOM  and  JTF,  although  there  was  no 
opportunity  to  assess  whether  these  command  levels  found  the  DFC2  schema  useful  and 
meaningful. 


Priority  Requirement  22  -  Provide  tools  that  enable  structured  Knowledge  Management 
publishing  processes  that  enable  producers  to  publish  their  products  as  XML-based  documents. 
[COI-2] 
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CINC  21  Technology  -  Enterprise  C2  Infrastructure 

Gj  Assessment  -  Three  critical  products  in  the  infrastructure  provide  tools  for  XML 

indexing  and  publishing.  Knowledge  Board  provides  tools  (Extensible  Data  Services  - 
XDS)  that  provide  document  and  web  page  XML  service  along  with  persistence.  XIS 
server  tools  provide  geo-spatial  indexing  and  publishing  tools  for  XML.  ORACLE®  8i/9i 
servers  provide  the  platfonn  for  actual  persistence.  All  are  operational  on  the  SIPRNET 
and  used  in  exercises. 


Priority  Requirement  23  -  Build  a  knowledge  portal  for  viewing  and  navigating  through  the 
COCOM’s  Ops-Intel  Brief,  SITREPs,  and  Electronic  Battlebook.  [COI-1] 

CINC  21  Technology  -  JWID  Admin  Portal,  DFC2 

(q)  Assessment  -  The  JWID  Admin  Portal,  which  was  not  developed  further  after  JWID, 
allowed  users  to  view  and  navigate  the  COCOM’s  daily  brief.  This  capability  was  highly 
rated  by  users.  The  DFC2  allows  users  to  browse  and  link  to  documents  such  as 
SITREPS;  however,  the  process  is  very  cumbersome  and  time-consuming. 


Priority  Requirement  24  -  Provide  knowledge  portals  for  the  JTF  and  Allies  that  enable  easy 
access  to  COCOM-provided  support  services.  [COI-1] 


CINC  21  Technology  -  DFC2,  JWID  Portal 


/\f\  Assessment  -  The  DFC2  presentation  layer  portal  is  applicable  across  echelons.  It  can 
‘  ^  provide  access  to  all  support  services,  but  was  not  used  in  this  capacity.  A  fonn  of  the 

DFC2  portal  was  used  in  JWID  02  along  with  COINs  to  demonstrate  interoperability 
with  Coalition  Partners.  The  JWID  portal  was  administrative  in  nature  and  is  not 
currently  operational. 


Priority  Requirement  25  -  Implement  data  extraction  and  automated  assembly  mechanisms  for 
automating  significant  portions  of  the  COCOM’s  morning  brief.  [COI-2] 


CINC  21  Technology  -  DFC2,  COA  Matrix,  RFI  Manager,  Consequence  Management, 
TCCC,  Message  Tracker 


A  Assessment  -  The  CINC  2 1  Auto  Brief  technologies  were  slated  to  provide  these 
services.  The  Auto  Brief  was  not  developed,  and  the  DFC2  concept  was  designed  to 
provide  output  that  fulfills  many  of  the  briefing  functions  now  comprising  the  COCOM’s 
morning  brief.  Implementing  mechanisms  are  not  fully  developed. 


Priority  Requirement  26  -  Develop  automated  brief  preparation  capability  based  upon  content 
and  update  of  the  information/knowledge  base.  [COI-1] 

CINC  21  Technology  -  DFC2,  Consequence  Management 

Assessment  -  The  Consequence  Management  tool  helped  users  locate,  manipulate,  and 
update  information  needed  for  briefings.  Although  it  was  considered  to  have  very  good 
utility,  some  improvements  to  the  user  interface  are  needed  to  improve  the  tool’s 
usability.  The  capabilities  of  DFC2  to  support  briefings  are  more  limited  at  this  time 
since  it  is  heavily  text-based  and  the  Decision  Summary  feature  has  not  yet  been  fully 
implemented. 
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Priority  Requirement  27  -  Develop  consequence  management  and  targeted  briefing  process. 
[COI-1] 

CINC  21  Technology  -  DFC2,  Consequence  Management 


Assessment  -  While  the  Consequence  Management  tool  provided  the  capability  for 
creating,  managing,  retrieving,  and  displaying  information  used  in  briefings,  no  effort 
was  directed  toward  the  business  rules,  work  process,  or  training  necessary  to  incorporate 
this  tool  into  USPACOM  operations. 


Requirement  28  -  Develop  the  standard  conference  room  configuration  to  allow  command/staff 
organic  support  (CINC  21 -configured  conference  room). 


CINC  21  Technology  -  DCTS,  IWS 


Assessment  -  This  requirement  was  met  via  the  provision  of  virtual  conference  rooms 
that  are  successfully  used  in  DCTS  and  IWS. 


Requirement  29  -  Develop  DTDs  for  OPLANs  and  CONPLANs.  [COI-1] 

CINC  21  Technology  -  Enterprise  C2  Infrastructure;  Knowledge  Core  and  DFC2:  Decision 
Space  Data  Schema,  Schema  Management 

A  Assessment-  DTDs  were  not  implemented  in  the  conventional  sense.  These  are  web  page 
capabilities  to  standardize  active  server  pages.  CINC  21  focused  on  the  knowledge  Core 
and  DFC2  data  layer  schema.  This  schema  can  support  the  OPLANS  and  CONPLANS. 
These  schemas  are  operational  on  the  SIPRNET. 

Requirement  30  -  Develop  the  dynamic  TPFDD. 


CINC  21  Technology  -  none 


Assessment  -  This  requirement  was  not  addressed  by  CINC  2 1 . 


Requirement  31  -  Implement  mechanisms  for  automating  processes  consistent  with  the 
workflow  management  system  currently  under  development  at  USPACOM.  [COI-1] 

CINC  21  Technology  -  Task  Management  System  (TMS) 

Assessment  -  The  Task  Management  System  is  fully  operational  at  USPACOM.  While 
there  are  improvements  to  be  made  in  workflow,  it  meets  current  operational 
requirements. 


Requirement  32  -  Provide  next-generation  situation  awareness,  planning,  and  force  execution 
tools  to  the  COCOM  and  integrate  with  the  Knowledge  Management  environment.  [COI-4] 

CINC  21  Technology  -  DFC2,  COA  Matrix,  FBV,  AOR  Basing 

A  Assessment  -  While  the  FBV,  AOR  Basing,  and  COA  Matrix  tools  have  been  developed 
to  provide  users  with  situation  awareness  and  improve  planning  and  force  execution  tasks 
as  part  of  the  DFC2  concept,  they  are  not  currently  operationally  integrated  within  the 
knowledge  management  environment. 
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Requirement  33  -  Provide  the  infrastructure  that  enables  selected  portions  of  the  COCOM’s 
networks  to  support  prioritized  delivery  of  products  and  bit-streams. 


CINC  21  Technology  -  none 


Assessment  -  This  requirement  was  not  addressed  by  CINC  2 1 . 


Requirement  34  -  Field  advanced  protocols  to  significantly  reduce  traffic  latency  of  selected 
information  flows  through  theater  networks. 


CINC  21  Technology  -  none 


Assessment  -  This  requirement  was  not  addressed  by  CINC  2 1 . 


Requirement  35  -  Develop  a  creditable  solution  for  auto -generation  of  security  tags  for  archived 
data  products  based  upon  the  current  coalition  release  policy. 


CINC  21  Technology  -  none 


Assessment  -  This  requirement  was  not  addressed  by  CINC  2 1 . 


Requirement  36  -  Provide  high-resolution,  large-screen  displays  to  the  COCOM  that  support 
simultaneous  crises  and  day-to-day  operations.  [COI-1] 

CINC  21  Technology  -  USPACOM  Video  Wall,  USSTRATCOM  Knowledge  Wall 

\  Assessment  -  The  large  screen  that  was  provided  to  USPACOM  is  fully  operational  and 
is  used  continuously.  It  affords  limited  utility  in  the  JOC,  however,  because  of  several 
ergonomic  and  visibility  problems  as  well  as  its  limited  technical  capabilities.  A  different 
display  system,  such  as  that  adopted  at  USSTRATCOM,  and  an  improved  workspace 
layout  would  substantially  enhance  the  utility  of  the  large  screen  displays  for  crisis  and 
routine  operations. 


Requirement  37  -  Develop  mechanisms  for  dynamically  managing  the  pixel  real-estate  of  large 
displays  to  improve  comprehension.  [COI-1] 

CINC  21  Technology  -  USPACOM  Video  Wall,  USSTRATCOM  Knowledge  Wall 

©Assessment  -  Both  large  screen  displays,  at  USSTRATCOM  and  USPACOM  are  fully 
operational  and  used  continuously.  The  USSTRATCOM  Knowledge  Wall  was  effective 
in  supporting  simultaneous  crises  and  day-to-day  operations.  However,  the  USPACOM 
Video  Wall  affords  limited  utility  in  the  JOC  because  of  several  ergonomic  and  visibility 
problems  as  well  as  limited  technical  abilities. 


Requirement  38  -  Provide  alerting  mechanisms  that  auto-initiate  appropriate  visualization 
events.  [COI-4] 

CINC  21  Technology  -  DFC2,  FBV,  TCCC 

(  )  Assessment  -  While  alerting  and  adaptive  visualization  are  part  of  the  DFC2  concept, 

these  capabilities  were  only  minimally  implemented  in  the  current  version  of  this  tool  via 
the  status  displays  of  decision  points  and  CCIRs.  This  capability  could  be  incorporated 
into  FBV  and  TCCC,  but  it  does  not  currently  exist.  This  requirement  was  not  sufficiently 
addressed  to  allow  assessment. 
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Requirement  39  -  Implement  homogeneous  server-server  federations.  [COI-4] 

CINC  21  Technology  -  Enterprise  C2  Infrastructure;  Knowledge  Core  (XML  Data  Services) 
and  DFC2  Schema  management 

©  Assessment-  The  Enterprise  C2  Infrastructure  provides  homogeneous  server-server 
federations.  It  is  different  in  that  it  does  not  provide  updates  to  all  servers  (replication  of 
infonnation)  but  it  provides  XML  that  points  to  the  servers  of  interest.  This  capability 
exists  on  the  SIPRNET,  and  federations  of  CUSeeMe®  conference  servers  were 
employed  in  KB(X). 


Requirement  40  -  Develop  the  standard  mobile  client  workstation  configuration  and  software  to 
allow  command/staff  organic  support  (CINC  21 -configured  mobile  workstation). 


CINC  21  Technology  -  none 


Assessment  -  This  requirement  was  not  addressed  by  CINC  2 1 . 


Requirement  41  -  Provide  high-resolution,  large  displays  to  a  JTF  capable  of  supporting 
simultaneous  crisis  operations.  [COI-4] 

CINC  21  Technology  -  USSTRATCOM  Knowledge  Wall,  PV290 

©  Assessment  -  This  requirement  was  addressed  during  KB(X).  The  USSTRATCOM 
Knowledge  Wall  provided  a  high-resolution  large  screen  to  the  I  MEF  JTF  to  support 
simultaneous  crisis  operations  in  coordination  with  a  COCOM. 


Requirement  42  -  Provide  a  method  for  accessing  knowledge  objects  across  security  enclaves 
with  minimal  latency. 

CINC  21  Technology  -  none 

Assessment  -  This  requirement  was  not  addressed  by  CINC  21 . 

Requirement  43  -  Prepare  for  the  transition  from  Ipv4  to  Ipv6  by  performing  selected 
experimentation  and  risk  reduction  activities  on  selected  theater  networks. 

CINC  21  Technology  -  none 

Assessment  -  This  requirement  was  not  addressed  by  CINC  21 . 

Requirement  44  -  Provide  multi-modal  human-computer  interface  capabilities  (e.g.,  voice,  head¬ 
tracking,  gesture,  hand -pointing). 

CINC  21  Technology  -  none 

Assessment  -  This  requirement  was  not  addressed  by  CINC  2 1 . 
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So  What? 


The  previous  sections  have  summarized  the  findings  from  many  of  the  CINC  21  products, 
assessment  events,  and  requirements.  Much  detail  has  been  provided,  and  even  more  specifics 
are  provided  in  the  individual  product  assessment  reports  (see  Appendices).  Yet,  it  is  often 
difficult  to  see  general  patterns  concerning  military  utility  and  transition  potential  from  detailed 
results.  Therefore,  this  section  combines  these  data  to  support  general  conclusions  about  the 
CINC  2 1  products. 

Summary  ratings  of  the  products,  organized  by  technology  categories,  are  presented  first.  The 
extent  to  which  the  prioritized  requirements  have  been  satisfied  for  each  COI  is  then  provided. 
The  findings  have  been  aggregated  for  the  CINC  21  COIs.  Finally,  transition  of  CINC  21 
products  is  discussed. 


Technology  Categories 


Table  12  summarizes  the  composite  ratings  for  Suitability,  Usability,  and  Technical  for  each 
CINC  21  technology  category.  The  CINC  21  Operational  Manager  and  Development  Manager 
had  identified  the  products  that  related  to  each  of  the  COIs.  These  mappings  were  used  to 
associate  the  technology  categories  with  the  COIs.  In  cases  where  a  technology  category  applies 
to  more  than  one  COI,  different  aspects  were  evaluated.  For  example,  DFC2  involves  the 
presentation  of  information  (COI-1)  and  the  organization  and  management  of  that  information  in 
a  common  knowledge  core  (COI-2). 

Ratings  were  assigned  to  the  individual  technologies  in  the  Special  Projects  category;  however, 
these  technologies  are  not  slated  for  further  development  or  for  transition.  Therefore,  ratings  of 
these  individual  technologies  are  not  combined  to  a  technology  category  composite  score. 

Decision-Focused  Command  and  Control 

The  concept  of  DFC2 — integrated  applications  that  facilitate  the  definition,  visualization, 
analysis,  and  management  of  decision-making  activities  associated  with  joint  command  and 
control — was  well-received  by  users.  However,  there  is  currently  a  lack  of  functioning  links 
between  component  parts  and  between  decision  objects  within  component  parts.  The  document 
linking  and  management  services  in  their  current  stage  of  development  do  not  fulfill  users’ 
needs.  In  addition,  concerns  regarding  the  impact  of  DFC2  upon  workload  and  organizational 
procedures  remain. 
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Table  12.  Technology  category  composite  ratings  for  Suitability  (S),  Usability  (U), 
and  Technical  (T)  assessment  criteria  in  terms  of  Critical  Operational  Issues  (COI). 


Technology 

Category 

COM 

COI-2 

COI-3 

001-4 

s 

U 

T 

S 

U 

T 

S 

U 

T 

S 

U 

T 

Decision-Focused 
C2  [details] 

A 

A 

A 

A 

A 

A 

NA 

NA 

NA 

NA 

NA 

NA 

Ops  Package: 
COP  /  Geo-Spatial 
Visualization 

[details] 

© 

© 

A 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

Ops  Package: 
Briefing  / 
Information 
Summary 
[detailsl 

© 

© 

© 

© 

© 

A 

NA 

NA 

NA 

NA 

NA 

NA 

Ops  Package: 
Status  Tracking 
and  Linking 

[details] 

© 

A 

© 

A 

A 

A 

NA 

NA 

NA 

NA 

NA 

NA 

Ops  Package: 
Time-based  Event 
Management 
[details] 

© 

© 

© 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

Ops  Package: 
JWID  Coalition 
Interoperability 
Service  (COINS) 
[detailsl 

NA 

NA 

NA 

© 

A 

© 

NA 

NA 

NA 

NA 

NA 

NA 

Displays: 
Group  Displays 

[details] 

© 

© 

© 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

Displays: 

Workstation 

Displays 

[details] 

© 

© 

© 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

Collaboration 

[details] 

NA 

NA 

NA 

NA 

NA 

NA 

© 

A 

A 

NA 

NA 

NA 

Network 

Operations  Tools 
and  Services 
[detailsl 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

© 

A 

A 
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For  COI-1,  Suitability  was  rated  as  YELLOW  due  to  the  concerns  expressed  above  and  various 
layout  problems  that  impeded  decision-focused  visualization.  Specifically,  some  of  the  web  parts 
were  located  completely  off  the  screen,  requiring  users  to  scroll  horizontally  and  vertically  to 
access  this  information.  Unfortunately,  even  if  users  did  know  that  this  off-screen  information 
existed,  they  would  have  great  difficulty  in  noticing  changes  or  in  using  it  to  provide  integrated 
situation  awareness.  Redesign  of  the  screen  layout  is  needed  so  that  users  can  simultaneously 
view  all  the  important  DFC2  information  without  scrolling.  What  information  is  considered 
important  is  likely  to  differ  across  users,  and  in  some  cases,  multi-screen  workstations  will  be 
necessary  to  meet  their  task  needs. 

Suitability  was  also  rated  as  YELLOW  for  COI-2.  The  inflexibility  of  the  schema  in  the  current 
version  of  DFC2  was  troublesome  for  users,  who  work  in  a  highly  dynamic  information 
environment.  The  procedural  adaptations  necessary  to  match  the  operational  information  and 
situations  to  the  DFC2  schema  were  considered  cumbersome. 

Usability  problems  were  experienced  throughout  the  assessment  events,  especially  in  areas  of 
configuration  retention  and  navigation.  Much  of  the  detail  is  presented  as  text  and  tables,  which 
inhibits  rapid  understanding  and  visualization  of  the  infonnation.  Excessive  scrollbars  limit 
users’  ability  to  maneuver,  and  they  present  a  confusing  display.  Many  functions  require 
numerous  steps  as  well  as  interaction  with  multiple  dialog  boxes.  Also,  the  use  of  screen  real 
estate  is  less  than  optimal;  web  parts  are  located  partially  or  completely  off  the  screen.  As  a 
result  of  these  problems,  Usability  was  rated  as  YELLOW  for  COI-1  and  COI-2. 

A  slow  refresh  rate  and  some  instability  of  component  parts  occurred  throughout  the  assessment 
events.  These  problems  interfered  with  users  as  they  attempted  to  input  or  access  information. 
There  were  intennittent  crashes  of  DFC2  during  TF03;  these  crashes  were  associated  with  data 
losses.  Thus,  the  Technical  rating  was  YELLOW  for  COI-1.  The  Technical  rating  for  COI-2  was 
also  YELLOW.  The  lack  of  linking  between  web  parts  seriously  inhibited  the  users’  ability  to 
access  and  edit  data  between  views. 

Operational  Packages 

Common  Operational  Picture/Geo-Spatial  Visualization 

AORB  and  the  FBV  earned  a  Green  rating  in  Suitability,  as  users  reported  that  both  would  help 
them  achieve  their  operational  goals.  They  appreciated  the  concept  of  the  geo-spatial 
representation,  in  which  they  could  drill  down  and  gather  their  desired  content.  During  the  time 
of  assessment,  AORB  was  in  an  early  stage  of  development  and  the  databases  needed  to  support 
its  capability  were  not  available.  The  Technical  category,  therefore,  received  a  YELLOW  rating. 
Although  not  as  serious  an  issue,  FBV  was  also  rated  as  YELLOW  on  Technical  because  it  did  not 
yet  include  information  classified  secret,  which  users  said  was  needed  to  be  of  real  value.  Both 
technologies  rated  a  GREEN  in  Usability  because  users  were  able  to  successfully  navigate  the 
tools  and  reach  their  desired  information.  Along  the  way,  however,  there  were  problems 
encountered  in  workflow,  highlighting  a  need  for  future  usability  assistance. 
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Briefing/Information  Summary 

The  COA  Matrix  and  Consequence  Management  tools  appear  to  meet  their  operational 
requirements.  Operational  staff  and  decision-makers  found  these  products  to  be  useful  in 
providing  a  rapid  way  to  visualize  the  situation  and  in  managing  the  underlying  infonnation 
consistently.  As  a  result,  Suitability  was  rated  as  GREEN  for  COI-1  and  COI-2. 

The  COA  Matrix  was  assessed  as  GREEN  on  Usability,  while  Consequence  Management  was 
rated  as  YELLOW.  Since  the  usability  of  the  COA  Matrix  was  so  well  received  by  its  audience, 
the  overall  summary  assessment  of  Usability  was  rated  as  GREEN  for  COI-1  and  COI-2.  The 
usability  problems  with  Consequence  Management,  principally  concerning  information 
presentation  and  visualization,  should  be  addressed  before  further  development. 

For  COI-1,  the  briefing  and  information  summary  products  were  rated  as  GREEN,  since  they 
operated  reliably  and  effectively  in  presenting  information  to  users.  While  these  products  seemed 
to  manage  the  underlying  data  accurately  and  effectively,  the  Technical  rating  for  COI-2  was 
YELLOW  because  of  security  issues.  To  become  more  useful  in  operational  settings,  these 
products  will  need  to  give  access  to  multi-level  security  materials. 

Status  Tracking  and  Linking 

For  this  technology  category,  Suitability  for  COI-1  is  Green.  The  Message  Tracker,  RFI 
manager,  and  TeamApp  all  allow  the  user  to  visualize  complex  information  in  a  fonn  that  is 
relevant  to  their  tasking.  The  Suitability  rating  for  COI-2  is  YELLOW.  User  management  issues 
are  of  concern.  The  “fit”  of  Message  Tracker  and  RFI  manager  into  the  users’  work  processes  is 
questionable,  as  was  the  level  of  user  acceptance  of  these  technologies  to  help  them  manager 
their  messages  and  information  requests. 

Usability  for  COI-1  is  YELLOW.  Colors  and  formatting  that  do  not  meet  human-computer 
interface  standards  negatively  affect  user  navigation  and  readability.  For  COI-2,  TeamApp  is 
easy  to  use  and  its  interface  features  support  the  intended  functions.  However,  training 
requirements  and  memory  and  workload  demands  are  excessive  for  Message  Tracker  and  RFI 
Manager.  The  combined  rating  is  YELLOW. 

Technical  for  COI-1  is  Green.  Operations  of  these  technologies  were  reliable  in  presenting  their 
information  to  users.  COI-2  is  YELLOW,  as  refresh  and  download  rates  may  be  too  slow  for 
operational  use. 

Time-Based  Event  Management 

While  there  is  room  for  improvement  in  addressing  the  needs  of  individual  users,  the  Master 
Calendar  and  the  TMS  generally  meet  the  needs  of  their  users.  They  have  been  accepted  by  the 
users  and  are  currently  used  at  USPACOM.  Therefore,  Suitability  is  assessed  as  GREEN. 
Usability  was  rated  as  GREEN  since  the  products  were  considered  easy  to  use.  Both  products  are 
currently  in  use  and  function  reliably,  so  the  Technical  rating  was  GREEN. 

JWID  Coalition  Interoperability  Service  (COINS) 

Suitability  for  the  JWID  Coalition  Interoperability  Service  (COINS)  component  is  GREEN. 
COINS  provides  capabilities  of  value  for  integrating  information,  context,  and  rules  to  increase 
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understanding  and  improve  decision-making.  Usability  for  JWID  COINS  component  is  Yellow. 
The  search  process  required  much  duplication  of  effort,  and  the  fonnat  for  display  of  results  was 
not  well-designed.  The  inability  to  seamlessly  transfer  data  to  other  applications  was  also 
associated  with  duplication  of  effort.  Technical  issues  are  rated  Green.  Availability  of  the 
technology  and  accessibility  to  the  data  was  adequate. 

Displays 

Group  Displays 

The  USSTRATCOM  Knowledge  Wall  and  the  USPACOM  Video  Wall  earned  GREEN  scores 
across  the  board.  Both  walls  are  currently  in  use,  accounting  for  a  GREEN  in  the  Technical 
category.  Both  earned  a  GREEN  in  Suitability.  The  USSTRATCOM  Knowledge  Wall  effectively 
met  the  needs  of  the  senior  decision-makers  during  the  October  demonstration.  It  also  is  rated 
well  (GREEN)  on  Usability.  The  Knowledge  Wall  operators  were  able  to  manipulate  content  and 
present  in  a  way  that  was  relevant  and  legible.  However,  there  are  currently  some  unresolved 
issues  regarding  the  Video  Wall  with  respect  to  Usability,  Suitability,  and  Technical  that  may 
require  further  study.  The  success  of  the  USSTRATCOM  Knowledge  Wall,  though,  outweighs 
any  inadequacies  found  on  the  USPACOM  Video  Wall. 

Workstation  Displays 

This  technology  category  received  a  score  of  GREEN  in  Suitability,  Usability,  and  Technical. 

The  multiple,  side-by-side  flat  panel  monitors  represent  a  completely  workable  solution  that  can 
compensate  for  the  drawbacks  of  the  PV290. 

The  PV290  was  used  during  the  USSTRATCOM  October  event.  It  was  well-received  by  users 
and  performed  without  incident.  During  TF03,  at  USPACOM,  users  reported  their  three  flat- 
panel  display  workstation  was  essential  in  providing  users  a  continuous  view  of  the  various  web 
parts.  This  view  increased  the  usability  of  the  displays  and  helped  meet  their  operational 
requirements.  However,  they  found  the  workstation  to  be  only  effective  in  presenting  content  to 
the  user  sitting  directly  in  front  of  the  screens,  and  not  to  others  standing  around  the  display. 
There  were  also  numerous  complaints  that  the  font  was  too  small. 

Collaboration 

Distributed  users  found  high  value  in  sharing  documents,  briefs,  and  desktops.  They  considered 
that  collaboration  tools  provide  an  efficient  mechanism  for  the  exchange  of  information.  Users 
judged  that  collaborative  sessions  led  to  increased  situation  awareness.  Suitability  is  rated  Green. 

However,  chat  often  became  the  de  facto  collaboration  tool  due  to  bandwidth  restrictions.  Audio 
was  preferred  when  available;  video  was  often  unstable  and  considered  by  many  users  to  be 
redundant  and  unnecessary.  The  status  of  audio-video  and  of  session  membership  was  frequently 
difficult  to  discern  with  the  DCTS  technologies.  Feedback  to  the  user  regarding  success  of  data 
transmittals  was  not  always  adequate.  In  addition,  several  areas  of  the  IWS  computer  interface 
were  identified  as  requiring  modification  to  improve  ease  of  use.  Other  impediments  to  use  are 
lack  of  training,  lack  of  business  rules,  and  lack  of  interoperability  between  collaboration 
systems.  Usability  is  rated  YELLOW. 
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The  inability  of  DCTS  technologies  to  provide  stable  connectivity  is  an  obstacle  toward  reaching 
the  goal  of  synchronizing  decision-making  and  infonnation  management  among  distributed 
users.  In  addition  to  the  technical  aspects,  personnel’s  confidence  and  willingness  to  use  these 
collaboration  tools  is  negatively  impacted.  However,  the  ability  to  provide  stable  connectivity 
was  demonstrated  with  IWS  in  actual  operational  use.  Therefore,  the  Technical  rating  for  the 
CINC  21  collaborative  services  technology  category  is  Yellow. 

Network  Operations  Tools  and  Services 

Many  Network  Operations  Tools  and  Services  were  implemented.  The  users  valued  the  assets, 
finding  the  information  current  and  precise.  These  tools  give  the  network  managers  the  ability  to 
interact  with  other  management  tools,  thus  providing  the  users  with  a  needed  capability.  The 
portal  approach  resulted  in  flexibility  of  display  and  screen  space  usage  and  external  access  to 
NetOps  generated  data.  Suitability  is  rated  Green. 

The  users  found  the  Theater  C4ISR  Status  Grid  useful  for  such  tasks  as  changing  layout  and 
exchanging  infonnation.  However,  the  workload  was  not  decreased  and  the  learning  curve  was 
difficult.  Usability  is  rated  Yellow. 

Transmitted  information  did  not  increase  ability  to  view  and  monitor  network  operations.  There 
were  problems  with  the  implementation  and  integration  of  Status  Monitoring.  The  EICS  data 
visualization  needs  improvement  for  easier  comprehension.  Link  analysis  visualization  requires 
additional  filtering  capability.  The  Technical  rating  is  Yellow. 

While  the  JMUA  was  limited  to  the  data  collected  at  that  time,  all  of  the  NetOps  tools  are 
scheduled  for  transition  to  DFC2,  which  will  be  implemented  in  the  DISA  Network-Centric 
Enterprise  Services  (NCES)/Joint  Command  and  Control  (JC2)  efforts. 

The  NetOps  Network  Status  Monitoring  will  transition  to  the  USPACOM  TCCC.  The  AIDE 
visualizations  (EICS,  EPC,  and  Link  Analysis)  will  transition  to  the  DISA  Infonnation 
Superiority  Situational  Awareness  (ISSA)  Portal  initiative. 


Prioritized  Requirements 


Table  13  summarizes  how  well  CINC  21  products  met  the  prioritized  requirements.  The  CINC 
21  Operational  Manager  and  Development  Manager  identified  which  requirements  supported 
each  COI,  and  the  table  reflects  that  mapping.  A  few  of  the  requirements  did  not  match  any  of 
the  COIs  (10,  28,  30,  33,  34,  35,  40,  42,  43,  44).  Some  of  the  requirements  that  did  relate  to  a 
COI  were  not  sufficiently  addressed  by  CINC  21  products  to  permit  assessment. 

For  COI-1,  9  of  the  13  requirements  were  in  the  top  27  priority  items.  Most  of  these 
requirements  were  assessed  as  partially  met.  Standard  hardware  and  software  configurations 
have  been  installed  at  USPACOM  and  STRACOM  and  are  operating  effectively  (Requirements 
1,  36,  37).  CINC  21  has  also  successfully  developed  software  to  maintain  situation  awareness  by 
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accessing  data,  by  viewing  briefings,  and  by  managing  workflow  (Requirements  4,  5,  23,  3 1). 
Domain-independent  mechanisms  for  tailoring  the  presentation  and  for  displaying  information 
linkages  have  been  partially  implemented  by  DFC2  and  other  products,  but  expanded  capabilities 
are  needed  (Requirements  2,  3).  Good  initial  efforts  were  made  to  access  support  services  and  to 
automate  the  preparation  of  briefing  materials,  but  substantial  additional  work  will  be  needed  to 
provide  an  operational  product  (Requirements  24,  26,  27,  29). 

All  four  of  the  requirements  for  COI-2  were  in  the  priority  set  of  27.  Mechanisms  for  the  theater¬ 
wide  access  to  information  and  published  products  were  successfully  developed  (Requirements 
20,  22).  Initial  efforts  to  organize  information  around  user-defined  themes  were  provided  by 
DFC2,  but  additional  development  is  needed  to  support  cross-echelon  knowledge  exchange 
(Requirement  21).  The  data  extraction  and  automated  mechanisms  to  support  preparation  of 
regular  briefings  and  reports  have  been  demonstrated,  but  further  work  is  required  to  implement 
these  capabilities  fully  (Requirement  25). 

Of  the  10  requirements  for  COI-3,  9  were  in  the  top  27  priority  set.  DCTS  and  IWS  provided 
virtual  conference  rooms  to  promote  collaboration  (Requirement  28).  DCTS  was  able  to  operate 
on  unclassified  and  classified  networks  to  connect  distributed  teams,  but  the  quality  and 
reliability  of  the  connections  was  poor — limiting  the  utility  of  these  collaborative  sessions 
(Requirements  6,  1 1,  13,  14,  15,  19).  DFC2  provides  the  framework  for  a  thematic  collaborative 
system,  but  a  greater  degree  of  user  aiding/automation  is  needed  to  implement  this  requirement 
fully  (Requirement  7).  The  capability  to  automatically  establish  and  manage  collaborative 
sessions  based  on  workflow  and  operational  priorities  has  been  partially  implemented  in  DFC2, 
and  further  enhancements  of  this  capability  are  planned  (Requirement  17). 

Four  of  the  eight  requirements  for  COI-4  were  among  the  27  priority  items.  VPN  capabilities 
were  successfully  implemented  to  enable  collaborative  tools  to  operate  in  the  presence  of 
firewalls  and  to  connect  allied  commanders  (Requirements  12,  18,  39,  41).  Several  tools  have 
been  developed  that  support  situation  awareness,  planning,  and  execution  monitoring,  but  these 
products  need  to  be  integrated  more  fully  with  the  knowledge  management  environment 
(Requirements  9,  32). 

Note  that  the  requirements,  as  stated,  primarily  concern  technical  issues  rather  than  suitability  or 
usability  issues.  For  the  most  part,  they  presume  that  improved  operational  workflow  and 
decision-making  will  result  from  the  presence  of  these  technical  capabilities. 
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Table  13.  Ratings  for  CINC  21  products  in  terms  of  prioritized  requirements  and  COI. 


COM 

[  details  1 

COI-2 

[ details! 

COI-3 

(details! 

COI-4 

(details! 

Priority 

Reqmt 

Assessment 

Rating 

Priority 

Reqmt 

Assessment 

Rating 

Priority 

Reqmt 

Assessment 

Rating 

Priority 

Reqmt 

Assessment 

Rating 

1 

© 

20 

© 

6 

A 

8 

2 

A 

21 

A 

7 

A 

9 

A 

3 

A 

22 

© 

11 

A 

12 

© 

4 

A 

25 

A 

13 

A 

18 

© 

5 

© 

14 

A 

32 

A 

23 

© 

15 

A 

38 

24 

A 

16 

39 

© 

26 

A 

17 

A 

41 

© 

27 

A 

19 

A 

29 

A 

28 

© 

31 

© 

36 

© 

37 

© 

Note: 


© 

A 


Green  =  requirement  met, 

Yellow  =  requirement  partially  met, 


Red  =  major  obstacle  prevents  requirement  from  being  met, 


B&W  =  requirement  not  addressed. 
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Critical  Operational  Issues 


It  is  useful  to  aggregate  across  the  CINC  2 1  products  and  technology  categories  to  provide  an 
overall  assessment  of  how  well  the  COIs  were  met.  To  do  this,  it  was  necessary  to  determine 
weights  for  how  much  the  technology  categories  support  each  COI.  A  consensus  among  six 
members  of  the  CINC  2 1  assessment  team  produced  the  following  weights.  The  group 
detennined  that  differential  weights  were  appropriate,  since  some  of  the  technology  categories 
were  more  central  to  the  core  concepts  of  CINC  21  and  the  COIs.  A  four-point  ordinal  scale  was 
adopted  to  define  the  category  weights.  As  it  becomes  possible  to  incorporate  input  from  the 
CINC  21  Operational  Manager,  Technical  Manager,  Development  Manager,  Implementation 
Manager,  and  Transition  Manager,  these  weights  may  be  adjusted  to  reflect  the  broader 
consensus. 

•  Very  Significant  (weight  =  4) . Decision-Focused  Command  and  Control 

•  Significant  Element  (weight  =  3) . Ops  Package:  COP/Geo-Spatial  Visualization 

Ops  Package:  Briefing/Information  Summary 
Ops  Package:  JWID  COINS 
Collaboration 

Network  Operations  Tools  and  Services 

•  Important  Element  (weight  =  2) . Ops  Package:  Status  Tracking  and  Linking 

•  Contributing  Element  (weight  =  1) . Ops  Package:  Time-based  Event  Management 

Displays:  Group  Displays 
Displays:  Workstation  Displays 

The  overall  assessments  of  Suitability,  Usability,  and  Technical  for  each  COI  are  shown  in  Table 
14.  The  CINC  21  products  that  contribute  to  COI-1  were  generally  considered  to  help  decision¬ 
makers  and  staff  to  process,  digest,  and  assimilate  large  amounts  of  complex  information.  Most 
of  these  products  were  useful  for  the  operational  tasks  and  work  processes,  and  they  were 
relatively  easy  to  use.  Certainly,  several  enhancements  can  be  made  to  further  these  products, 
and  these  refinements  can  be  made  during  the  technology  transition  phase.  Suitability  and 
Usability  are  rated  as  Green  for  the  combination  of  products  related  to  COI-1.  The  overall  rating 
for  COI-1  for  Technical  is  YELLOW  because  many  of  the  products  were  not  yet  developed  to  the 
point  where  they  could  reliably  provide  dynamic,  real-world  data  to  users. 

Overall,  the  Suitability  and  Usability  COI-1  objectives  are  being  met,  but  there  are  concerns  with 
the  DFC2  that  should  be  addressed.  There  are  still  issues  to  be  resolved  regarding  how  best  to 
integrate  all  the  relevant  information  and  present  it  in  a  useful  manner. 

The  CINC  21  products  and  concepts  provided  many  exciting  and  innovative  approaches  to 
knowledge  management.  Technologies  that  help  integrate  information,  context,  and  rules  will 
certainly  increase  situation  understanding  and  improve  decision-making.  The  idea  of  providing  a 
framework  for  linking  and  integrating  information  is  an  excellent  step  towards  meeting  the  users’ 
operational  requirements.  The  current  versions  of  the  products,  however,  need  further  refinement 
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to  achieve  these  goals  to  the  levels  needed  by  operational  users.  For  COI-2,  Suitability,  Usability, 
and  Technical  ratings  were  all  YELLOW.  Suitability  can  be  improved  by  linking  more  of  the  key 
data  sources  together.  Usability  can  be  improved  by  making  it  easier  for  operational  users  to 
ingest  and  link  messages  and  other  data  together.  As  discussed  above,  many  technical  challenges 
remain  to  be  resolved  concerning  how  to  enable  CINC  2 1  products  to  deliver  the  range  of 
operational  content  that  users  need  in  an  accurate  and  reliable  manner. 

Table  14.  Overall  assessments  of  Suitability,  Usability,  and  Technical  for  each  COI. 


Critical  Operational  Issue 

Suitability 

Usability 

Technical 

1  -  Decision-Focused 
Visualization 

© 

© 

A 

2  -  Knowledge  Management 

A 

A 

A 

3  -  Collaboration 

© 

A 

A 

4  -  Enterprise  Awareness  & 
Network  Security 

© 

A 

A 

As  called  for  in  COI-3,  collaboration  technology  is  an  important  capability  that  can  enhance 
distributed  decision-making  substantially.  Users  consistently  reinforced  this,  reporting  that  the 
CINC  2 1  collaboration  products  were  very  useful  for  their  operational  needs.  Suitability,  was, 
therefore,  rated  as  Green.  Some  of  the  collaboration  products  were  difficult  to  configure  and  to 
use,  however.  This  difficulty  accounts  for  the  YELLOW  rating  for  Usability.  The  biggest  obstacle 
for  collaboration  technology  was  clearly  the  Technical  issues;  this  area  was  also  rated  as 
YELLOW.  The  products  were  often  incompatible  with  each  other,  required  excessive  bandwidth, 
and  operated  unreliably.  If  these  technical  issues  can  be  overcome,  then  collaboration  technology 
would  add  an  important  capability. 

COI-4  calls  for  advances  in  the  secure  exchange  of  information  across  multiple,  distributed 
networks,  databases,  and  applications.  The  CINC  2 1  products  developed  to  satisfy  COI-4  include 
Theater  C4ISR  Status  Grid  and  Network  Operations  Status  Monitoring  Tools.  Security  via  VPN 
(as  part  of  the  Special  Projects)  was  enabled  and  provided  secure  collaboration  capabilities. 
Network  tools  allowed  ready  access  to  data  across  the  coalition  network.  Given  these  successes, 
as  well  as  the  homogenous  server-server  federations  and  flexibility,  the  Network  Operations  tool 
suite  will  be  featured  in  next-generation  implementations.  Suitability  for  COI-4  was  rated  as 
Green.  There  were  some  operational  and  integration  issues,  however,  with  information  assurance 
tools  that  impede  achievement  of  all  requirements.  Ratings  for  Usability  and  Technical  were 
Yellow. 
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Transition  Recommendations 


Many  varied  types  of  technical  products  were  developed  as  part  of  the  CINC  2 1  program.  While 
some  applications  directly  support  users’  work  processes  and  tasks,  others  involve  system 
architecture  concepts  and  enterprise  infrastructure  that  enable  other  products  to  function 
effectively.  Some  products  explore  the  feasibility  of  potentially  relevant  techniques,  and  other 
interim  products  were  abandoned  as  the  focus  changed  due  to  external  factors,  like  NMCI. 

As  a  result,  transition  recommendations  are  somewhat  difficult  to  provide  for  all  CINC  2 1 
products.  Our  approach  focuses  on  products  that  operational  decision-makers  and  staff  would  use 
directly.  The  underlying  concepts  and  infrastructure  that  enable  users  to  use  these  products  are 
incorporated  implicitly.  Interim  products  and  special  projects  are  not  considered. 

CINC  21  Products  for  Transition 

The  set  of  CINC  21  products  that  are  candidates  for  transition  according  to  this  approach  were 
identified.  The  Suitability,  Usability,  and  Technical  assessment  for  these  products  were  reviewed 
to  determine  which  products  are  mature  enough  for  transition  and  have  adequate  potential  to 
provide  military  utility.  These  transition  products  are  listed  below  in  Table  15  along  with  their 
summary  ratings  for  Suitability,  Usability,  and  Technical.  Specific  enhancements  are 
recommended  that  are  intended  to  help  these  products  achieve  an  acceptable  level  of 
perfonnance  in  an  operational  setting.  These  recommendations  are  presented  in  more  detail 
following  the  table. 

The  order  in  which  the  products  appear  in  Table  15  reflects  the  Operational  Manager’s  priorities, 
which  were  largely  determined  by  the  amount  of  effort  estimated  to  be  required  to  make  the 
recommended  enhancements.  Thus,  products  toward  the  top  of  the  list  are  anticipated  to  reach 
transition  sooner  than  those  toward  the  bottom. 

TeamApp 

This  web-based  product  helps  users  organize  information  that  is  considered  pertinent  to 
an  event.  It  was  found  to  be  useful  for  transferring  files,  sharing  briefings,  responding  to 
CCIRs,  and  exchanging  information.  Users  considered  TeamApp  easy  to  use,  and  they 
used  it  for  various  purposes.  Improved  system  response  time  for  downloads  would  be  a 
significant  enhancement,  however. 

•  Reduce  the  download  time,  particularly  under  limited  bandwidth  conditions. 
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Table  15.  CINC  21  products  recommended  for  transition 


CINC  21  Product 

Suitability 

Usability 

Technical 

Recommended  Enhancements 

TeamApp 

Tdetailsl 

© 

© 

A 

•  Reduce  download  times 

Task 

Management 

System 

Tdetailsl 

A 

© 

© 

•  Provide  user  training  and  notification 

•  Show  task  dependencies 

Master  Calendar 

Tdetailsl 

© 

© 

© 

•  Add  flexible  calendar  configurations 

Fused 

Battlespace 

View 

Tdetailsl 

© 

© 

A 

•  Allow  info  exchange  across  security  levels 

RFI  Manager 

Tdetailsl 

A 

A 

© 

•  Integrate  COCOM  and  JTF  processes 

•  Improve  the  user  interface 

•  Allow  custom  displays 

•  Provide  user  training  and  on-line  help 

Network 
Operations 
Tools  and 
Services 

Tdetailsl 

© 

A 

A 

•  Integrate  with  work  processes 

•  Reduce  the  number  of  task  steps  needed 
to  operate 

•  Integrate  with  current  systems 

Visualization 

(Displays) 

Tdetailsl 

© 

© 

© 

•  Determine  layout  of  information  on  multi¬ 
screen  workspaces 

Message 

Tracker 

Tdetailsl 

© 

A 

© 

•  Integrate  with  other  tools 

•  Improve  the  user  interface 

•  Provide  user  training  and  on-line  help 

DFC2 

Tdetailsl 

A 

A 

A 

•  Develop  expanded  capabilities 

•  Simplify  /  automate  message  ingestion 

•  Improve  the  user  interface 

•  Provide  user  training  and  on-line  help 

Collaboration 

Tdetailsl 

© 

A 

A 

•  Provide  user  training  and  business  rules 

•  Provide  interoperability 

•  Improve  system  reliability 

130 


Joint  Military  Utility  Assessment 


Task  Management  System 

This  product  provides  a  web-based  structure  for  tracking  events  and  tasks  across  offices 
and  echelons.  The  system  has  been  in  operational  use  for  some  time  without  incident,  and 
the  latest  version  has  good  usability.  The  only  significant  limitations  with  this  product 
concerns  who  is  using  the  system  and  how  they  are  using  it.  User  training  and  standard 
operating  procedures  would  be  helpful  here. 

•  Provide  training  and  standard  operating  procedures  so  that  all  users  use  the  system 
in  a  similar  manner  and  for  similar  purposes. 

•  Develop  a  task  notification  procedure/technology  to  supplement  e-mail,  since 
personnel  currently  need  to  be  at  their  computer  to  receive  tasking. 

•  Provide  a  way  to  show  task  dependencies  and  conflicts. 

Master  Calendar 

This  product  is  a  web-based  tool  that  presents  time-based  events  in  a  calendar  format.  It 
provides  several  different  views,  enabling  users  to  detect  conflicts  and  overlaps.  It  was 
very  useful  and  easy  to  use.  It  has  been  operating  effectively  at  USPACOM  for  some 
time.  Only  minor  enhancements  were  recommended. 

•  Add  more  options  for  customizing  the  display  to  allow  flexible  calendar 
configurations. 

Fused  Battlespace  View 

This  product  was  developed  to  access  and  manipulate  information  relevant  for 
USSTRATCOM.  It  provides  a  geo-spatial  context  with  drill-down  capability  within 
USSTRATCOM  operational  process.  Users  found  it  very  useful,  reducing  the  time  and 
personnel  involved  in  retrieving  information.  Users  found  it  easy  to  use,  but  they  needed 
access  to  multi-level  security  information 

•  Provide  access  to  unclassified  and  classified  infonnation. 

•  Expand  the  drill-down  capability  to  include  additional  levels. 

•  Improve  some  aspects  of  the  user  interface,  including  font  size,  link  colors,  etc. 

RFI  Manager 

RFI  Manager  is  a  stand-alone  product  tracks  requests  for  information,  due  dates, 
completion  status,  and  responsible  office/person.  It  has  been  used  extensively  in  the  JTF 
environment,  but  RFI  Manager  has  not  yet  been  fully  integrated  with  USPACOM 
systems.  While  the  RFI  Manager  is  a  useful  tool,  particularly  at  JTFs,  it  needs  to  be 
integrated  with  other  systems,  and  its  user  interface  needs  various  upgrades. 

•  Integrate  RFI  Manager  with  COCOM  and  JTF  processes  and  databases. 

•  Conduct  a  job/workflow  analysis  to  incorporate  RFI  Manager  into  COCOM  and 
JTF  message  management  processes  without  increasing  workload. 

•  Improve  the  user  interface  in  several  ways,  including  navigation  aids,  feedback, 
error  checking,  alerting,  screen  layout,  RFI  status  summary,  etc. 
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•  Provide  the  capability  for  users  to  customize  their  displays. 

•  Provide  user  training,  documentation,  and  online  help. 

•  Increase  refresh  rate  to  a  level  appropriate  for  operational  use. 

Network  Operations  Tools  and  Services 

This  product  currently  represents  a  suite  of  tools  intended  to  help  users  view  the 
operational  status  of  networks  in  the  theater.  These  tools  were  generally  considered 
useful  for  presenting  important  status  information  for  theater  networks  and  information 
assurance.  The  visualizations  were  well-suited  for  TCCC  users  but  were  less  directly 
useful  for  other  potential  users,  such  as  those  in  the  JOC.  Several  enhancements  in  the 
usability  and  technical  aspects  of  these  tools  are  recommended. 

•  Revise  the  Theater  C4ISR  Status  Grid  display  to  provide  infonnation  in  a  form 
that  is  more  closely  integrated  with  TCCC  and  JOC  work  processes. 

•  Reduce  the  number  of  task  steps  needed  to  operate  the  Theater  C4ISR  Status 
Grid,  and  simplify  the  user  interface  to  make  it  easier  to  learn  and  use. 

•  Reformat  the  information  presented  in  the  EICS  and  link  analysis  tools  to  improve 
user  comprehension  of  the  data  and  to  reduce  display  complexity/clutter. 

•  Implement  these  tools  in  an  operational  environment  (e.g.,  USPACOM)  to 
demonstrate  reliable  access,  rapid  system  response  time,  and  accurate  data 
transfer  with  current  systems. 

Visualization  (Displays) 

Several  display  products  were  examined  to  provide  improved  visualization  of  complex 
data  sets.  Displays  included  large  screens  designed  for  group  viewing,  as  well  as  multi¬ 
screen  workstations  for  individual  users.  While  some  products  were  found  to  be 
unsuitable  for  COCOM  and  JTF  usage,  others  met  the  operational  needs  quite  well. 

These  display  technologies  were  considered  to  be  useful,  easy  to  use,  and  stable.  Careful 
consideration  of  the  layout  of  information  to  be  displayed  on  these  devices  is  needed, 
however. 

•  Develop  default  settings  that  allow  the  displayed  information  to  be  properly 
scaled,  sized,  and  configured  to  support  the  expected  number  of  viewers  (e.g., 
large  group,  small  team,  individual)  under  work  area  and  lighting  constraints. 

•  Develop  guidelines  for  the  effective  layout  of  information  onto  the  available 
screen  space.  Provide  user  training  to  encourage  display  customization. 

Message  Tracker 

This  product  allows  users  to  read  messages  and  link  them  to  operations  or  to  CCIRs.  It 
has  various  capabilities  for  capturing,  filtering,  grouping,  archiving,  and  organizing 
messages.  It  was  found  to  be  effective  at  linking  messages  to  significant  events  and 
increasing  staff  access  to  relevant  information.  Several  usability  enhancements  are 
recommended,  but  the  Message  Tracker  has  been  functioning  effectively  in  the 
USPACOM  environment. 
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•  Integrate  Message  Tracker  with  current  tools  and  work  processes. 

•  Improve  the  user  interface  in  several  ways,  including  navigation  aids,  background 
colors,  feedback,  error  checking,  alerting,  and  display  customization. 

•  Provide  user  training,  documentation,  and  online  help. 

•  Increase  the  refresh  rate  to  a  level  appropriate  for  operational  use. 

Decision-Focused  Command  Control  (DFC2) 

This  is  the  main  product  of  the  CINC  21  effort.  DFC2  is  a  set  of  software  components 
that  facilitate  an  array  of  information  management  functions.  It  is  implemented  as  a 
presentation  layer,  business  layer,  and  data  layer.  Users  considered  the  DFC2  concept  to 
be  valuable  for  managing  large,  complex  data  sets  and  for  maintaining  situation 
awareness  of  multiple  events.  The  current  implementation  of  DFC2,  however,  needs 
substantially  greater  functionality  to  become  suitable  for  operational  use.  Similarly, 
several  usability  and  technical  problems  need  to  be  overcome  before  DFC2  can  achieve 
military  utility. 

•  Provide  pedigree  information  about  objects  and  information  elements,  including 
state  changes,  user  comments,  alerts,  context  changes,  and  links  to  other  tasks. 

•  Integrate  collaboration  tools  with  objects  and  information  elements.  This 
integration  includes  capabilities  for  text  chat,  shared  data,  task  linking,  and  shared 
audio/video. 

•  Provide  an  easy-to-use  capability  for  alerting  users  based  on  context-sensitive  and 
position-sensitive  rules  and  conditions. 

•  Develop  labels  and  schema  that  match  COCOM  processes  and  terms. 

•  Enable  persistence  of  data  displays  for  individual  users. 

•  Provide  error  checking  support,  date-time  conversion,  and  automatic  backup. 

•  Enable  messages  to  be  parsed  and  ingested  into  the  system  automatically. 

•  Implement  the  capability  to  link  messages  and  other  information  products  to  other 
objects,  processes,  collaboration  workspaces,  CCIRs,  etc. 

•  Develop  easy-to-use  rules  for  status  thresholds  and  for  displaying  status 
information. 

•  Provide  capabilities  for  searching,  fdtering,  and  organizing  messages,  documents, 
and  other  information. 

•  Improve  the  user  interface  in  numerous  ways,  including  layout  of  infonnation  on 
screens,  font  size,  text  wrap,  scrolling,  highlights,  colors,  icons,  consistency,  etc. 

•  Improve  data  visualization,  particularly  for  decision  summaries,  by  enabling  data 
to  be  shown  in  various  forms  (tabular,  geo-spatial,  temporal,  etc.). 

•  Enable  the  user  to  associate  information  objects  via  drag-and-drop. 
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•  Provide  the  capability  for  personalization/customization  of  user  information 
displays  within  the  framework  of  predefined  defaults  and  templates. 

•  Develop  substantially  expanded  capabilities  for  creating,  editing,  deleting, 
importing,  linking,  and  manipulating  objects  and  information  elements. 

•  Provide  the  capability  to  navigate  through  the  decision  space  hierarchically, 
temporally,  and  geo-spatially  depending  on  the  situation  and  user  needs. 

•  Develop  capabilities  for  assigning,  tracking,  and  managing  tasks  within  an 
integrated  DFC2  framework. 

•  Provide  user  training,  documentation,  on-line  aids,  and  templates. 

•  Assure  reliable  system  operation  under  distributed  operations  with  multiple 
concurrent  users. 


Collaboration 

The  general  capability  for  collaboration  relies  on  several  tools,  including  DCTS  and  IWS. 
Users  considered  collaboration  tools  to  be  highly  useful,  promoting  shared  situation 
awareness.  Various  usability  issues  and  operating  procedures  should  be  addressed  to 
improve  information  exchange  among  multiple  concurrent  users.  The  most  substantial 
shortcoming,  however,  concerns  the  unstable  connectivity,  limited  interoperability  among 
different  collaboration  systems,  and  high  bandwidth  consumption. 

•  Improve  the  feedback  provided  to  users  concerning  login,  active/available 
sessions,  active  participants,  data  transfers,  and  status  of  audio/video  channels. 

•  Provide  training,  business  rules,  or  standard  operating  procedures  for  efficient  use 
of  collaboration  tools  and  sessions. 

•  Provide  stable  connectivity  among  an  operationally  relevant  number  of  concurrent 
users  on  the  full  range  of  collaboration  tools. 

•  Provide  reliable  interoperability  among  different  collaboration  systems. 

Several  CINC  2 1  products  were  identified  as  not  ready  for  transition.  These  products  are 
listed  below.  Some  of  these  products  were  developed  only  as  interim  products  to  test  the 
feasibility  of  a  particular  solution,  while  others  were  not  pursued  for  various  technical 
reasons. 

•  Area  of  Responsibility  Basing 

•  Consequence  Management  Automated  Brief 

•  Course  of  Action  Matrix 

•  JWID  COINS 

•  USPACOM  Video  Wall 

•  PV290  Multi-Screen  Desktop  Display 

•  BMC  Patrol 

•  USSTRATCOM  Portal 

•  JWID  Administrative  Portal 

•  Quality  of  Service 
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•  Virtual  Private  Network 

•  3Com  B>  Embedded  Firewall  NIC  Cards 

•  Other  Special  Projects  (IPv6,  NETIQ,  NET  VCR,  MRTG) 


Extended  User  Evaluation 

To  move  forward  to  an  effective  transition  of  CINC  21  products,  several  important  steps  are 
recommended.  Together,  these  steps  are  part  of  the  Extended  User  Evaluation  (EUE). 


The  CINC  21  products  that  have  been  identified  for  transition  need  to  be  incorporated  into  a 
feasible  standard  operating  procedure  (SOP)  that  includes  defined  staff  roles  and  work  processes. 
While  substantial  variations  in  the  details  of  the  SOP  are  expected  across  COCOMs  because  of 
their  unique  responsibilities  and  operating  areas,  there  should  be  a  common  framework.  The 
CINC  2 1  operating  concepts  and  system  architecture  enable  the  products  to  be  used  in  a  common 
manner  that  generalizes  to  many  different  environments,  including  COCOMs  and  JTFs. 


Various  enhancements  will  need  to  be  made  to  the  CINC  21  products  to  make  them  more 
suitable,  usable,  and  technically  sound  in  operational  settings.  Several  specific  suggestions  for 
these  enhancements  have  been  provided  for  CINC  21  products  in  this  report  and  in  previous 
assessment  reports  (see  Appendices).  Operational  users  should  have  periodic  opportunities  to 
review  the  technical  modifications  and  to  provide  feedback  on  the  enhancements. 

Once  the  product  enhancements  have  been  made  and  reviewed  by  representative  operational 
users,  a  variety  of  supporting  documentation  will  need  to  be  prepared.  This  documentation 
includes  training  materials,  system  support  and  administration  procedures,  and  similar  reference 
materials. 


Finally,  the  enhanced  product  capabilities  should  be  assessed  in  a  realistic  operational  context. 
This  assessment  is  not  focused  on  determining  military  utility  but  rather  on  verifying  that  the 
product  enhancements  function  properly  and  do  indeed  satisfy  user  needs  and  system 
requirements. 
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B 
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FY  02-03.  San  Diego:  Space  &  Naval 
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CINC  21 
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Heacox,  N.  (Feb  03).  CINC  21 
Technologies:  Final  Status  of  Data 
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Engineering  Group,  Inc. 
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D 
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of  CINC  21  technology: 

The  decision-focused  command  and 
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Chat 
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The  decision-focused  command  and 
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Henry,  C.  (May  2002).  Review  of  the 
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Outcome 
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Henry,  C.  (Jul  2002).  A  Limited 
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Henry,  C.  (May  2002).  Review  of  the 
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